INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) INFO: 172.20.0.1:53586 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48658 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60386 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60400 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60416 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37488 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54902 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55350 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37370 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39498 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48892 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48894 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48900 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60920 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42542 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48088 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51916 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39194 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38376 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38378 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38386 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55272 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36420 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43040 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43886 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51016 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45884 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45898 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45908 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58180 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57804 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:17:40,902 - lava_callback - INFO - Received callback for node 69c71e1129015d0f9598f72b 2026-03-30 00:17:40,909 - lava_callback - INFO - Starting async job submit for node 69c71e1129015d0f9598f72b INFO: 172.20.0.1:44976 - "POST /node/69c71e1129015d0f9598f72b HTTP/1.0" 202 Accepted 2026-03-30 00:17:40,911 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:17:40,934 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e1129015d0f9598f72b HTTP/1.1" 200 None 2026-03-30 00:17:40,935 - lava_callback - DEBUG - Processing node 69c71e1129015d0f9598f72b: baseline-arm64-baylibre 2026-03-30 00:17:40,936 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c71e1129015d0f9598f72b 2026-03-30 00:17:40,937 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:17:40,959 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e1129015d0f9598f72b HTTP/1.1" 200 None 2026-03-30 00:17:40,961 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:17:40,976 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 00:17:40,977 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:17:41,008 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c71e1129015d0f9598f72b HTTP/1.1" 200 None 2026-03-30 00:17:41,015 - lava_callback - ERROR - Error processing callback for node 69c71e1129015d0f9598f72b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44982 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40320 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44798 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59124 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59136 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59152 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60760 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45342 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37498 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60198 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:19:00,897 - lava_callback - INFO - Received callback for node 69c71e6829015d0f9598f770 INFO: 172.20.0.1:32988 - "POST /node/69c71e6829015d0f9598f770 HTTP/1.0" 202 Accepted 2026-03-30 00:19:00,903 - lava_callback - INFO - Starting async job submit for node 69c71e6829015d0f9598f770 2026-03-30 00:19:00,904 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:19:00,923 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e6829015d0f9598f770 HTTP/1.1" 200 None 2026-03-30 00:19:00,924 - lava_callback - DEBUG - Processing node 69c71e6829015d0f9598f770: baseline-arm64-baylibre 2026-03-30 00:19:00,925 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c71e6829015d0f9598f770 2026-03-30 00:19:00,925 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:19:00,939 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e6829015d0f9598f770 HTTP/1.1" 200 None 2026-03-30 00:19:00,940 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:19:00,962 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 00:19:00,963 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:19:00,983 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c71e6829015d0f9598f770 HTTP/1.1" 200 None 2026-03-30 00:19:00,984 - lava_callback - ERROR - Error processing callback for node 69c71e6829015d0f9598f770: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:32994 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58454 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58460 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36058 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38840 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51734 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50676 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37878 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54812 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54814 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54822 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46740 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43658 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:20:41,032 - lava_callback - INFO - Received callback for node 69c71ec529015d0f9598f78c INFO: 172.20.0.1:35866 - "POST /node/69c71ec529015d0f9598f78c HTTP/1.0" 202 Accepted 2026-03-30 00:20:41,037 - lava_callback - INFO - Starting async job submit for node 69c71ec529015d0f9598f78c 2026-03-30 00:20:41,038 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:20:41,065 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71ec529015d0f9598f78c HTTP/1.1" 200 None 2026-03-30 00:20:41,066 - lava_callback - DEBUG - Processing node 69c71ec529015d0f9598f78c: baseline-arm64-baylibre 2026-03-30 00:20:41,066 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c71ec529015d0f9598f78c 2026-03-30 00:20:41,067 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:20:41,093 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71ec529015d0f9598f78c HTTP/1.1" 200 None 2026-03-30 00:20:41,095 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:20:41,129 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 00:20:41,131 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:20:41,156 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c71ec529015d0f9598f78c HTTP/1.1" 200 None 2026-03-30 00:20:41,158 - lava_callback - ERROR - Error processing callback for node 69c71ec529015d0f9598f78c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35874 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43432 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48388 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52816 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52818 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52830 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33232 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43206 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48968 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35838 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:22:01,069 - lava_callback - INFO - Received callback for node 69c71f1b29015d0f9598f797 INFO: 172.20.0.1:59918 - "POST /node/69c71f1b29015d0f9598f797 HTTP/1.0" 202 Accepted 2026-03-30 00:22:01,075 - lava_callback - INFO - Starting async job submit for node 69c71f1b29015d0f9598f797 2026-03-30 00:22:01,076 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:22:01,104 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71f1b29015d0f9598f797 HTTP/1.1" 200 None 2026-03-30 00:22:01,105 - lava_callback - DEBUG - Processing node 69c71f1b29015d0f9598f797: baseline-arm64-baylibre 2026-03-30 00:22:01,105 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c71f1b29015d0f9598f797 2026-03-30 00:22:01,105 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:22:01,119 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71f1b29015d0f9598f797 HTTP/1.1" 200 None 2026-03-30 00:22:01,121 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:22:01,138 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 00:22:01,140 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:22:01,187 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c71f1b29015d0f9598f797 HTTP/1.1" 200 None 2026-03-30 00:22:01,188 - lava_callback - ERROR - Error processing callback for node 69c71f1b29015d0f9598f797: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59932 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57604 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57616 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57624 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43030 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34750 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41586 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47380 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49668 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59604 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59616 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59618 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:23:21,164 - lava_callback - INFO - Received callback for node 69c71f7229015d0f9598f7a4 INFO: 172.20.0.1:41158 - "POST /node/69c71f7229015d0f9598f7a4 HTTP/1.0" 202 Accepted 2026-03-30 00:23:21,170 - lava_callback - INFO - Starting async job submit for node 69c71f7229015d0f9598f7a4 2026-03-30 00:23:21,171 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:23:21,207 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71f7229015d0f9598f7a4 HTTP/1.1" 200 None 2026-03-30 00:23:21,208 - lava_callback - DEBUG - Processing node 69c71f7229015d0f9598f7a4: baseline-arm64-baylibre 2026-03-30 00:23:21,208 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c71f7229015d0f9598f7a4 2026-03-30 00:23:21,208 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:23:21,236 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71f7229015d0f9598f7a4 HTTP/1.1" 200 None 2026-03-30 00:23:21,237 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:23:21,250 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 00:23:21,252 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:23:21,273 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c71f7229015d0f9598f7a4 HTTP/1.1" 200 None 2026-03-30 00:23:21,278 - lava_callback - ERROR - Error processing callback for node 69c71f7229015d0f9598f7a4: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41164 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37900 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60208 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55778 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50084 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33634 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33640 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33652 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54074 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60556 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:24:41,188 - lava_callback - INFO - Received callback for node 69c71fc329015d0f9598f7ae INFO: 172.20.0.1:44586 - "POST /node/69c71fc329015d0f9598f7ae HTTP/1.0" 202 Accepted 2026-03-30 00:24:41,193 - lava_callback - INFO - Starting async job submit for node 69c71fc329015d0f9598f7ae 2026-03-30 00:24:41,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:24:41,208 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71fc329015d0f9598f7ae HTTP/1.1" 200 None 2026-03-30 00:24:41,209 - lava_callback - DEBUG - Processing node 69c71fc329015d0f9598f7ae: baseline-arm64-baylibre 2026-03-30 00:24:41,209 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c71fc329015d0f9598f7ae 2026-03-30 00:24:41,210 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:24:41,227 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71fc329015d0f9598f7ae HTTP/1.1" 200 None 2026-03-30 00:24:41,229 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:24:41,246 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 00:24:41,248 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:24:41,268 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c71fc329015d0f9598f7ae HTTP/1.1" 200 None 2026-03-30 00:24:41,274 - lava_callback - ERROR - Error processing callback for node 69c71fc329015d0f9598f7ae: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44598 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45734 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52684 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58306 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58320 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58336 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50178 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53860 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47672 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50818 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39988 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42458 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42468 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42478 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:26:23,877 - lava_callback - INFO - Received callback for node 69c7201e29015d0f9598f7bc INFO: 172.20.0.1:40634 - "POST /node/69c7201e29015d0f9598f7bc HTTP/1.0" 202 Accepted 2026-03-30 00:26:23,883 - lava_callback - INFO - Starting async job submit for node 69c7201e29015d0f9598f7bc 2026-03-30 00:26:23,884 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:26:23,900 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7201e29015d0f9598f7bc HTTP/1.1" 200 None 2026-03-30 00:26:23,901 - lava_callback - DEBUG - Processing node 69c7201e29015d0f9598f7bc: baseline-arm64-clabbe 2026-03-30 00:26:23,902 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7201e29015d0f9598f7bc 2026-03-30 00:26:23,902 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:26:23,916 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7201e29015d0f9598f7bc HTTP/1.1" 200 None 2026-03-30 00:26:23,917 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:26:23,931 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 00:26:23,933 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:26:23,973 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7201e29015d0f9598f7bc HTTP/1.1" 200 None 2026-03-30 00:26:23,974 - lava_callback - ERROR - Error processing callback for node 69c7201e29015d0f9598f7bc: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:40646 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54456 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39808 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36498 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48210 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34978 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34980 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34992 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51632 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39548 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:27:43,907 - lava_callback - INFO - Received callback for node 69c7207329015d0f9598f7cb INFO: 172.20.0.1:36902 - "POST /node/69c7207329015d0f9598f7cb HTTP/1.0" 202 Accepted 2026-03-30 00:27:43,914 - lava_callback - INFO - Starting async job submit for node 69c7207329015d0f9598f7cb 2026-03-30 00:27:43,914 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:27:43,930 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7207329015d0f9598f7cb HTTP/1.1" 200 None 2026-03-30 00:27:43,931 - lava_callback - DEBUG - Processing node 69c7207329015d0f9598f7cb: baseline-arm64-clabbe 2026-03-30 00:27:43,931 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7207329015d0f9598f7cb 2026-03-30 00:27:43,932 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:27:43,947 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7207329015d0f9598f7cb HTTP/1.1" 200 None 2026-03-30 00:27:43,949 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:27:43,966 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 00:27:43,968 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:27:44,005 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7207329015d0f9598f7cb HTTP/1.1" 200 None 2026-03-30 00:27:44,011 - lava_callback - ERROR - Error processing callback for node 69c7207329015d0f9598f7cb: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:36904 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60676 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49502 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52040 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52056 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52068 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43416 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60240 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60158 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44460 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:29:04,009 - lava_callback - INFO - Received callback for node 69c720c629015d0f9598f7dc INFO: 172.20.0.1:40884 - "POST /node/69c720c629015d0f9598f7dc HTTP/1.0" 202 Accepted 2026-03-30 00:29:04,018 - lava_callback - INFO - Starting async job submit for node 69c720c629015d0f9598f7dc 2026-03-30 00:29:04,019 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:29:04,037 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c720c629015d0f9598f7dc HTTP/1.1" 200 None 2026-03-30 00:29:04,038 - lava_callback - DEBUG - Processing node 69c720c629015d0f9598f7dc: baseline-arm64-clabbe 2026-03-30 00:29:04,038 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c720c629015d0f9598f7dc 2026-03-30 00:29:04,039 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:29:04,059 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c720c629015d0f9598f7dc HTTP/1.1" 200 None 2026-03-30 00:29:04,061 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:29:04,077 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 00:29:04,079 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:29:04,126 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c720c629015d0f9598f7dc HTTP/1.1" 200 None 2026-03-30 00:29:04,127 - lava_callback - ERROR - Error processing callback for node 69c720c629015d0f9598f7dc: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:40886 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41540 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41546 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41548 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39526 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49504 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55456 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41096 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45294 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52544 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52564 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44260 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41838 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:30:44,113 - lava_callback - INFO - Received callback for node 69c7212329015d0f9598f7ea INFO: 172.20.0.1:49524 - "POST /node/69c7212329015d0f9598f7ea HTTP/1.0" 202 Accepted 2026-03-30 00:30:44,123 - lava_callback - INFO - Starting async job submit for node 69c7212329015d0f9598f7ea 2026-03-30 00:30:44,124 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:30:44,156 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7212329015d0f9598f7ea HTTP/1.1" 200 None 2026-03-30 00:30:44,158 - lava_callback - DEBUG - Processing node 69c7212329015d0f9598f7ea: baseline-arm64-clabbe 2026-03-30 00:30:44,158 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7212329015d0f9598f7ea 2026-03-30 00:30:44,158 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:30:44,181 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7212329015d0f9598f7ea HTTP/1.1" 200 None 2026-03-30 00:30:44,183 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:30:44,207 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 00:30:44,209 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:30:44,256 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7212329015d0f9598f7ea HTTP/1.1" 200 None 2026-03-30 00:30:44,276 - lava_callback - ERROR - Error processing callback for node 69c7212329015d0f9598f7ea: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49526 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35050 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58266 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36996 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37002 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37012 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51524 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41582 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40082 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:32:04,117 - lava_callback - INFO - Received callback for node 69c7217629015d0f9598f7fa INFO: 172.20.0.1:41854 - "POST /node/69c7217629015d0f9598f7fa HTTP/1.0" 202 Accepted 2026-03-30 00:32:04,123 - lava_callback - INFO - Starting async job submit for node 69c7217629015d0f9598f7fa 2026-03-30 00:32:04,124 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:32:04,138 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7217629015d0f9598f7fa HTTP/1.1" 200 None 2026-03-30 00:32:04,139 - lava_callback - DEBUG - Processing node 69c7217629015d0f9598f7fa: baseline-arm64-clabbe 2026-03-30 00:32:04,139 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7217629015d0f9598f7fa 2026-03-30 00:32:04,140 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:32:04,163 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7217629015d0f9598f7fa HTTP/1.1" 200 None 2026-03-30 00:32:04,165 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:32:04,180 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 00:32:04,182 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:32:04,203 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7217629015d0f9598f7fa HTTP/1.1" 200 None 2026-03-30 00:32:04,204 - lava_callback - ERROR - Error processing callback for node 69c7217629015d0f9598f7fa: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41870 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41906 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41922 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41934 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37608 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45938 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57154 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44068 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57738 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55476 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-30 00:33:16,500 - lava_callback - INFO - Received callback for node 69c9c4533c3aafdab4be9d1c INFO: 172.20.0.1:55492 - "POST /node/69c9c4533c3aafdab4be9d1c HTTP/1.0" 202 Accepted 2026-03-30 00:33:16,506 - lava_callback - INFO - Starting async job submit for node 69c9c4533c3aafdab4be9d1c INFO: 172.20.0.1:55508 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:33:16,528 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:33:16,543 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c4533c3aafdab4be9d1c HTTP/1.1" 200 None 2026-03-30 00:33:16,544 - lava_callback - DEBUG - Processing node 69c9c4533c3aafdab4be9d1c: baseline-arm 2026-03-30 00:33:16,733 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:33:16,909 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:33:16,910 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c4533c3aafdab4be9d1c/log.txt.gz 2026-03-30 00:33:17,020 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:33:17,278 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:33:17,280 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c4533c3aafdab4be9d1c/lava-logs.yaml 2026-03-30 00:33:17,285 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:33:17,456 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:33:17,458 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c4533c3aafdab4be9d1c/lava_callback.json.gz 2026-03-30 00:33:17,481 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:33:17,506 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c4533c3aafdab4be9d1c HTTP/1.1" 200 None 2026-03-30 00:33:17,508 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:33:17,521 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1213c3aafdab4be774c HTTP/1.1" 200 None 2026-03-30 00:33:17,523 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:33:17,563 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c4533c3aafdab4be9d1c HTTP/1.1" 200 None 2026-03-30 00:33:17,564 - lava_callback - INFO - Completed processing callback for node 69c9c4533c3aafdab4be9d1c INFO: 172.20.0.1:55516 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:33:24,191 - lava_callback - INFO - Received callback for node 69c721cb29015d0f9598f80a INFO: 172.20.0.1:46640 - "POST /node/69c721cb29015d0f9598f80a HTTP/1.0" 202 Accepted 2026-03-30 00:33:24,196 - lava_callback - INFO - Starting async job submit for node 69c721cb29015d0f9598f80a 2026-03-30 00:33:24,197 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:33:24,212 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c721cb29015d0f9598f80a HTTP/1.1" 200 None 2026-03-30 00:33:24,213 - lava_callback - DEBUG - Processing node 69c721cb29015d0f9598f80a: baseline-arm64-clabbe 2026-03-30 00:33:24,213 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c721cb29015d0f9598f80a 2026-03-30 00:33:24,214 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:33:24,228 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c721cb29015d0f9598f80a HTTP/1.1" 200 None 2026-03-30 00:33:24,230 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:33:24,242 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 00:33:24,244 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:33:24,264 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c721cb29015d0f9598f80a HTTP/1.1" 200 None 2026-03-30 00:33:24,272 - lava_callback - ERROR - Error processing callback for node 69c721cb29015d0f9598f80a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42146 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:33:41,018 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:33:41,038 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:53452 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40164 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38416 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40134 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40144 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40158 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54332 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47620 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:34:41,752 - lava_callback - INFO - Received callback for node 69c7221329015d0f9598f813 INFO: 172.20.0.1:56664 - "POST /node/69c7221329015d0f9598f813 HTTP/1.0" 202 Accepted 2026-03-30 00:34:41,758 - lava_callback - INFO - Starting async job submit for node 69c7221329015d0f9598f813 2026-03-30 00:34:41,759 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:34:41,787 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7221329015d0f9598f813 HTTP/1.1" 200 None 2026-03-30 00:34:41,788 - lava_callback - DEBUG - Processing node 69c7221329015d0f9598f813: baseline-arm64-baylibre 2026-03-30 00:34:41,788 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7221329015d0f9598f813 2026-03-30 00:34:41,789 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:34:41,812 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7221329015d0f9598f813 HTTP/1.1" 200 None 2026-03-30 00:34:41,814 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:34:41,833 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 00:34:41,835 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:34:41,880 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7221329015d0f9598f813 HTTP/1.1" 200 None 2026-03-30 00:34:41,882 - lava_callback - ERROR - Error processing callback for node 69c7221329015d0f9598f813: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:56678 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57882 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39766 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60950 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60958 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60960 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43930 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35232 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37272 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41892 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46336 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34576 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34586 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34590 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38624 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56680 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:36:41,815 - lava_callback - INFO - Received callback for node 69c7228b29015d0f9598f825 INFO: 172.20.0.1:50948 - "POST /node/69c7228b29015d0f9598f825 HTTP/1.0" 202 Accepted 2026-03-30 00:36:41,821 - lava_callback - INFO - Starting async job submit for node 69c7228b29015d0f9598f825 2026-03-30 00:36:41,822 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:36:41,841 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7228b29015d0f9598f825 HTTP/1.1" 200 None 2026-03-30 00:36:41,842 - lava_callback - DEBUG - Processing node 69c7228b29015d0f9598f825: baseline-arm64-baylibre 2026-03-30 00:36:41,842 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7228b29015d0f9598f825 2026-03-30 00:36:41,842 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:36:41,856 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7228b29015d0f9598f825 HTTP/1.1" 200 None 2026-03-30 00:36:41,858 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:36:41,871 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 00:36:41,872 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:36:41,892 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7228b29015d0f9598f825 HTTP/1.1" 200 None 2026-03-30 00:36:41,899 - lava_callback - ERROR - Error processing callback for node 69c7228b29015d0f9598f825: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:50956 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34630 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37878 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44156 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44158 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44168 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42924 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44318 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55600 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45782 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38556 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53270 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53280 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53288 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47774 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55308 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38976 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57972 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:39:01,983 - lava_callback - INFO - Received callback for node 69c7231429015d0f9598f87a INFO: 172.20.0.1:59532 - "POST /node/69c7231429015d0f9598f87a HTTP/1.0" 202 Accepted 2026-03-30 00:39:01,989 - lava_callback - INFO - Starting async job submit for node 69c7231429015d0f9598f87a 2026-03-30 00:39:01,990 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:39:02,011 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7231429015d0f9598f87a HTTP/1.1" 200 None 2026-03-30 00:39:02,011 - lava_callback - DEBUG - Processing node 69c7231429015d0f9598f87a: baseline-arm-baylibre 2026-03-30 00:39:02,012 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7231429015d0f9598f87a 2026-03-30 00:39:02,012 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:39:02,025 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7231429015d0f9598f87a HTTP/1.1" 200 None 2026-03-30 00:39:02,027 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:39:02,040 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 00:39:02,042 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:39:02,061 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7231429015d0f9598f87a HTTP/1.1" 200 None 2026-03-30 00:39:02,063 - lava_callback - ERROR - Error processing callback for node 69c7231429015d0f9598f87a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59546 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53162 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53172 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53186 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43894 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45566 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:39:42,011 - lava_callback - INFO - Received callback for node 69c7233929015d0f9598f890 INFO: 172.20.0.1:42282 - "POST /node/69c7233929015d0f9598f890 HTTP/1.0" 202 Accepted 2026-03-30 00:39:42,021 - lava_callback - INFO - Starting async job submit for node 69c7233929015d0f9598f890 2026-03-30 00:39:42,023 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:39:42,057 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7233929015d0f9598f890 HTTP/1.1" 200 None 2026-03-30 00:39:42,065 - lava_callback - DEBUG - Processing node 69c7233929015d0f9598f890: baseline-arm-baylibre 2026-03-30 00:39:42,065 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7233929015d0f9598f890 2026-03-30 00:39:42,068 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:39:42,094 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7233929015d0f9598f890 HTTP/1.1" 200 None 2026-03-30 00:39:42,096 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:39:42,121 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 00:39:42,124 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:39:42,182 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7233929015d0f9598f890 HTTP/1.1" 200 None 2026-03-30 00:39:42,184 - lava_callback - ERROR - Error processing callback for node 69c7233929015d0f9598f890: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:42288 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:39:52,009 - lava_callback - INFO - Received callback for node 69c9c6383c3aafdab4be9d52 INFO: 172.20.0.1:41220 - "POST /node/69c9c6383c3aafdab4be9d52 HTTP/1.0" 202 Accepted 2026-03-30 00:39:52,015 - lava_callback - INFO - Starting async job submit for node 69c9c6383c3aafdab4be9d52 2026-03-30 00:39:52,033 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:39:52,048 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6383c3aafdab4be9d52 HTTP/1.1" 200 None 2026-03-30 00:39:52,048 - lava_callback - DEBUG - Processing node 69c9c6383c3aafdab4be9d52: baseline-x86 2026-03-30 00:39:52,266 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:39:52,623 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:39:52,625 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69c9c6383c3aafdab4be9d52/log.txt.gz 2026-03-30 00:39:52,732 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:39:53,161 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:39:53,164 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69c9c6383c3aafdab4be9d52/lava-logs.yaml 2026-03-30 00:39:53,170 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:39:53,345 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:39:53,347 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69c9c6383c3aafdab4be9d52/lava_callback.json.gz 2026-03-30 00:39:53,365 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:39:53,397 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6383c3aafdab4be9d52 HTTP/1.1" 200 None 2026-03-30 00:39:53,399 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:39:53,416 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c13f3c3aafdab4be7771 HTTP/1.1" 200 None 2026-03-30 00:39:53,418 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:39:53,443 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6383c3aafdab4be9d52 HTTP/1.1" 200 None 2026-03-30 00:39:53,445 - lava_callback - INFO - Completed processing callback for node 69c9c6383c3aafdab4be9d52 INFO: 172.20.0.1:41224 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:39:59,517 - lava_callback - INFO - Received callback for node 69c9c6363c3aafdab4be9d49 INFO: 172.20.0.1:41240 - "POST /node/69c9c6363c3aafdab4be9d49 HTTP/1.0" 202 Accepted 2026-03-30 00:39:59,525 - lava_callback - INFO - Starting async job submit for node 69c9c6363c3aafdab4be9d49 2026-03-30 00:39:59,544 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:39:59,560 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6363c3aafdab4be9d49 HTTP/1.1" 200 None 2026-03-30 00:39:59,561 - lava_callback - DEBUG - Processing node 69c9c6363c3aafdab4be9d49: baseline-x86 2026-03-30 00:39:59,880 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:40:00,108 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:40:00,110 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69c9c6363c3aafdab4be9d49/log.txt.gz 2026-03-30 00:40:00,223 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:40:00,823 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:40:00,825 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69c9c6363c3aafdab4be9d49/lava-logs.yaml 2026-03-30 00:40:00,831 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:40:00,978 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:40:00,980 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69c9c6363c3aafdab4be9d49/lava_callback.json.gz 2026-03-30 00:40:01,001 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:40:01,017 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6363c3aafdab4be9d49 HTTP/1.1" 200 None 2026-03-30 00:40:01,019 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:40:01,033 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c13f3c3aafdab4be7771 HTTP/1.1" 200 None 2026-03-30 00:40:01,034 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:40:01,082 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6363c3aafdab4be9d49 HTTP/1.1" 200 None 2026-03-30 00:40:01,084 - lava_callback - INFO - Completed processing callback for node 69c9c6363c3aafdab4be9d49 INFO: 172.20.0.1:60238 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:40:11,042 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:40:11,060 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:41708 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41712 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41726 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56622 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56458 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59664 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59816 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:40:59,327 - lava_callback - INFO - Received callback for node 69c9c6373c3aafdab4be9d4f INFO: 172.20.0.1:59822 - "POST /node/69c9c6373c3aafdab4be9d4f HTTP/1.0" 202 Accepted 2026-03-30 00:40:59,333 - lava_callback - INFO - Starting async job submit for node 69c9c6373c3aafdab4be9d4f 2026-03-30 00:40:59,358 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:40:59,403 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6373c3aafdab4be9d4f HTTP/1.1" 200 None 2026-03-30 00:40:59,404 - lava_callback - DEBUG - Processing node 69c9c6373c3aafdab4be9d4f: baseline-x86 2026-03-30 00:40:59,818 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:41:00,039 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:41:00,041 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69c9c6373c3aafdab4be9d4f/log.txt.gz 2026-03-30 00:41:00,217 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:41:00,690 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:41:00,692 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69c9c6373c3aafdab4be9d4f/lava-logs.yaml 2026-03-30 00:41:00,700 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:41:00,850 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:41:00,852 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69c9c6373c3aafdab4be9d4f/lava_callback.json.gz 2026-03-30 00:41:00,870 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:00,885 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6373c3aafdab4be9d4f HTTP/1.1" 200 None 2026-03-30 00:41:00,886 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:00,901 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c13f3c3aafdab4be7771 HTTP/1.1" 200 None 2026-03-30 00:41:00,902 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:00,930 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6373c3aafdab4be9d4f HTTP/1.1" 200 None 2026-03-30 00:41:00,932 - lava_callback - INFO - Completed processing callback for node 69c9c6373c3aafdab4be9d4f INFO: 172.20.0.1:59834 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:41:11,062 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:11,079 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:46708 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46718 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46728 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:41:22,200 - lava_callback - INFO - Received callback for node 69c723b029015d0f9598f8cd INFO: 172.20.0.1:44442 - "POST /node/69c723b029015d0f9598f8cd HTTP/1.0" 202 Accepted 2026-03-30 00:41:22,205 - lava_callback - INFO - Starting async job submit for node 69c723b029015d0f9598f8cd 2026-03-30 00:41:22,207 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:22,225 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c723b029015d0f9598f8cd HTTP/1.1" 200 None 2026-03-30 00:41:22,227 - lava_callback - DEBUG - Processing node 69c723b029015d0f9598f8cd: baseline-arm-baylibre 2026-03-30 00:41:22,227 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c723b029015d0f9598f8cd 2026-03-30 00:41:22,228 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:22,244 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c723b029015d0f9598f8cd HTTP/1.1" 200 None 2026-03-30 00:41:22,246 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:22,260 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 00:41:22,262 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:22,318 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c723b029015d0f9598f8cd HTTP/1.1" 200 None 2026-03-30 00:41:22,320 - lava_callback - ERROR - Error processing callback for node 69c723b029015d0f9598f8cd: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44452 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:41:27,604 - lava_callback - INFO - Received callback for node 69c9c6363c3aafdab4be9d4c INFO: 172.20.0.1:44460 - "POST /node/69c9c6363c3aafdab4be9d4c HTTP/1.0" 202 Accepted 2026-03-30 00:41:27,610 - lava_callback - INFO - Starting async job submit for node 69c9c6363c3aafdab4be9d4c 2026-03-30 00:41:27,630 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:27,644 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6363c3aafdab4be9d4c HTTP/1.1" 200 None 2026-03-30 00:41:27,645 - lava_callback - DEBUG - Processing node 69c9c6363c3aafdab4be9d4c: baseline-x86 2026-03-30 00:41:28,064 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:41:28,325 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:41:28,327 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69c9c6363c3aafdab4be9d4c/log.txt.gz 2026-03-30 00:41:28,560 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:41:28,794 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:41:28,796 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69c9c6363c3aafdab4be9d4c/lava-logs.yaml 2026-03-30 00:41:28,805 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:41:28,946 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:41:28,948 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69c9c6363c3aafdab4be9d4c/lava_callback.json.gz 2026-03-30 00:41:28,967 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:28,981 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6363c3aafdab4be9d4c HTTP/1.1" 200 None 2026-03-30 00:41:28,983 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:28,996 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c13f3c3aafdab4be7771 HTTP/1.1" 200 None 2026-03-30 00:41:28,997 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:29,023 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6363c3aafdab4be9d4c HTTP/1.1" 200 None 2026-03-30 00:41:29,031 - lava_callback - INFO - Completed processing callback for node 69c9c6363c3aafdab4be9d4c 2026-03-30 00:41:30,132 - lava_callback - INFO - Received callback for node 69c9c6a13c3aafdab4be9d9f INFO: 172.20.0.1:43378 - "POST /node/69c9c6a13c3aafdab4be9d9f HTTP/1.0" 202 Accepted 2026-03-30 00:41:30,138 - lava_callback - INFO - Starting async job submit for node 69c9c6a13c3aafdab4be9d9f 2026-03-30 00:41:30,141 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:30,157 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6a13c3aafdab4be9d9f HTTP/1.1" 200 None 2026-03-30 00:41:30,158 - lava_callback - DEBUG - Processing node 69c9c6a13c3aafdab4be9d9f: baseline-arm 2026-03-30 00:41:30,163 - lava_callback - WARNING - Failed to upload log 2026-03-30 00:41:30,166 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:41:30,300 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:41:30,303 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c6a13c3aafdab4be9d9f/lava-logs.yaml 2026-03-30 00:41:30,305 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:41:30,480 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:41:30,482 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c6a13c3aafdab4be9d9f/lava_callback.json.gz 2026-03-30 00:41:30,485 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:30,500 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6a13c3aafdab4be9d9f HTTP/1.1" 200 None 2026-03-30 00:41:30,501 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:30,515 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:41:30,517 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:30,537 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6a13c3aafdab4be9d9f HTTP/1.1" 200 None 2026-03-30 00:41:30,561 - lava_callback - INFO - Completed processing callback for node 69c9c6a13c3aafdab4be9d9f INFO: 172.20.0.1:43388 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:41:41,081 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:41,098 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:36096 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:41:49,669 - lava_callback - INFO - Received callback for node 69c9c6a23c3aafdab4be9da2 INFO: 172.20.0.1:36108 - "POST /node/69c9c6a23c3aafdab4be9da2 HTTP/1.0" 202 Accepted 2026-03-30 00:41:49,718 - lava_callback - INFO - Starting async job submit for node 69c9c6a23c3aafdab4be9da2 2026-03-30 00:41:49,736 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:49,764 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6a23c3aafdab4be9da2 HTTP/1.1" 200 None 2026-03-30 00:41:49,765 - lava_callback - DEBUG - Processing node 69c9c6a23c3aafdab4be9da2: baseline-arm 2026-03-30 00:41:49,894 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:41:50,013 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:41:50,015 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c6a23c3aafdab4be9da2/log.txt.gz 2026-03-30 00:41:50,107 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:41:50,256 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:41:50,258 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c6a23c3aafdab4be9da2/lava-logs.yaml 2026-03-30 00:41:50,262 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:41:50,383 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:41:50,385 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c6a23c3aafdab4be9da2/lava_callback.json.gz 2026-03-30 00:41:50,405 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:50,420 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6a23c3aafdab4be9da2 HTTP/1.1" 200 None 2026-03-30 00:41:50,422 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:50,435 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:41:50,437 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:50,471 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6a23c3aafdab4be9da2 HTTP/1.1" 200 None 2026-03-30 00:41:50,472 - lava_callback - INFO - Completed processing callback for node 69c9c6a23c3aafdab4be9da2 2026-03-30 00:41:51,543 - lava_callback - INFO - Received callback for node 69c9c6a43c3aafdab4be9dab INFO: 172.20.0.1:53262 - "POST /node/69c9c6a43c3aafdab4be9dab HTTP/1.0" 202 Accepted 2026-03-30 00:41:51,590 - lava_callback - INFO - Starting async job submit for node 69c9c6a43c3aafdab4be9dab 2026-03-30 00:41:51,607 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:51,621 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6a43c3aafdab4be9dab HTTP/1.1" 200 None 2026-03-30 00:41:51,622 - lava_callback - DEBUG - Processing node 69c9c6a43c3aafdab4be9dab: baseline-arm 2026-03-30 00:41:51,756 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:41:51,868 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:41:51,870 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c6a43c3aafdab4be9dab/log.txt.gz 2026-03-30 00:41:51,945 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:41:52,104 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:41:52,106 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c6a43c3aafdab4be9dab/lava-logs.yaml 2026-03-30 00:41:52,110 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:41:52,223 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:41:52,225 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c6a43c3aafdab4be9dab/lava_callback.json.gz 2026-03-30 00:41:52,242 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:52,258 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6a43c3aafdab4be9dab HTTP/1.1" 200 None 2026-03-30 00:41:52,259 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:52,273 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:41:52,275 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:41:52,299 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6a43c3aafdab4be9dab HTTP/1.1" 200 None 2026-03-30 00:41:52,305 - lava_callback - INFO - Completed processing callback for node 69c9c6a43c3aafdab4be9dab INFO: 172.20.0.1:53270 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34986 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:42:11,100 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:42:11,117 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:50014 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50028 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:42:16,891 - lava_callback - INFO - Received callback for node 69c9c6a33c3aafdab4be9da8 INFO: 172.20.0.1:50030 - "POST /node/69c9c6a33c3aafdab4be9da8 HTTP/1.0" 202 Accepted 2026-03-30 00:42:16,898 - lava_callback - INFO - Starting async job submit for node 69c9c6a33c3aafdab4be9da8 2026-03-30 00:42:16,920 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:42:16,935 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6a33c3aafdab4be9da8 HTTP/1.1" 200 None 2026-03-30 00:42:16,936 - lava_callback - DEBUG - Processing node 69c9c6a33c3aafdab4be9da8: baseline-arm 2026-03-30 00:42:17,223 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:42:17,366 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:42:17,368 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c6a33c3aafdab4be9da8/log.txt.gz 2026-03-30 00:42:17,499 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:42:17,684 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:42:17,686 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c6a33c3aafdab4be9da8/lava-logs.yaml 2026-03-30 00:42:17,692 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:42:17,928 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:42:17,930 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c6a33c3aafdab4be9da8/lava_callback.json.gz 2026-03-30 00:42:17,954 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:42:17,972 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6a33c3aafdab4be9da8 HTTP/1.1" 200 None 2026-03-30 00:42:17,974 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:42:17,989 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:42:17,991 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.20.0.1:50036 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:42:18,072 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6a33c3aafdab4be9da8 HTTP/1.1" 200 None 2026-03-30 00:42:18,074 - lava_callback - INFO - Completed processing callback for node 69c9c6a33c3aafdab4be9da8 INFO: 172.20.0.1:37794 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58762 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:42:41,119 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:42:41,135 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:54880 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51426 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:42:56,670 - lava_callback - INFO - Received callback for node 69c9c6b13c3aafdab4be9dcd INFO: 172.20.0.1:51434 - "POST /node/69c9c6b13c3aafdab4be9dcd HTTP/1.0" 202 Accepted 2026-03-30 00:42:56,677 - lava_callback - INFO - Starting async job submit for node 69c9c6b13c3aafdab4be9dcd 2026-03-30 00:42:56,699 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:42:56,728 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6b13c3aafdab4be9dcd HTTP/1.1" 200 None 2026-03-30 00:42:56,729 - lava_callback - DEBUG - Processing node 69c9c6b13c3aafdab4be9dcd: baseline-arm-broonie 2026-03-30 00:42:56,937 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:42:57,069 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:42:57,071 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c9c6b13c3aafdab4be9dcd/log.txt.gz 2026-03-30 00:42:57,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:42:57,393 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:42:57,395 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c9c6b13c3aafdab4be9dcd/lava-logs.yaml 2026-03-30 00:42:57,401 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:42:57,540 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:42:57,542 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c9c6b13c3aafdab4be9dcd/lava_callback.json.gz 2026-03-30 00:42:57,565 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:42:57,584 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6b13c3aafdab4be9dcd HTTP/1.1" 200 None 2026-03-30 00:42:57,586 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:42:57,609 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:42:57,610 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:42:57,634 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6b13c3aafdab4be9dcd HTTP/1.1" 200 None 2026-03-30 00:42:57,644 - lava_callback - INFO - Completed processing callback for node 69c9c6b13c3aafdab4be9dcd 2026-03-30 00:42:58,940 - lava_callback - INFO - Received callback for node 69c9c6b23c3aafdab4be9dd2 INFO: 172.20.0.1:51442 - "POST /node/69c9c6b23c3aafdab4be9dd2 HTTP/1.0" 202 Accepted 2026-03-30 00:42:58,946 - lava_callback - INFO - Starting async job submit for node 69c9c6b23c3aafdab4be9dd2 2026-03-30 00:42:58,969 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:42:58,993 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6b23c3aafdab4be9dd2 HTTP/1.1" 200 None 2026-03-30 00:42:58,994 - lava_callback - DEBUG - Processing node 69c9c6b23c3aafdab4be9dd2: baseline-arm-broonie 2026-03-30 00:42:59,232 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:42:59,349 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:42:59,351 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c9c6b23c3aafdab4be9dd2/log.txt.gz 2026-03-30 00:42:59,476 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:42:59,653 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:42:59,655 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c9c6b23c3aafdab4be9dd2/lava-logs.yaml 2026-03-30 00:42:59,660 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:42:59,797 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:42:59,799 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c9c6b23c3aafdab4be9dd2/lava_callback.json.gz 2026-03-30 00:42:59,824 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:42:59,840 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6b23c3aafdab4be9dd2 HTTP/1.1" 200 None 2026-03-30 00:42:59,842 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:42:59,867 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:42:59,869 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:42:59,894 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6b23c3aafdab4be9dd2 HTTP/1.1" 200 None 2026-03-30 00:42:59,896 - lava_callback - INFO - Completed processing callback for node 69c9c6b23c3aafdab4be9dd2 2026-03-30 00:43:05,966 - lava_callback - INFO - Received callback for node 69c9c6d63c3aafdab4be9e4d INFO: 172.20.0.1:43816 - "POST /node/69c9c6d63c3aafdab4be9e4d HTTP/1.0" 202 Accepted 2026-03-30 00:43:06,013 - lava_callback - INFO - Starting async job submit for node 69c9c6d63c3aafdab4be9e4d 2026-03-30 00:43:06,033 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:06,055 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6d63c3aafdab4be9e4d HTTP/1.1" 200 None 2026-03-30 00:43:06,056 - lava_callback - DEBUG - Processing node 69c9c6d63c3aafdab4be9e4d: baseline-arm-pengutronix 2026-03-30 00:43:06,167 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:43:06,281 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:43:06,282 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9c6d63c3aafdab4be9e4d/log.txt.gz 2026-03-30 00:43:06,344 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:43:06,498 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:43:06,500 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9c6d63c3aafdab4be9e4d/lava-logs.yaml 2026-03-30 00:43:06,503 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:43:06,623 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:43:06,625 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9c6d63c3aafdab4be9e4d/lava_callback.json.gz 2026-03-30 00:43:06,646 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:06,661 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6d63c3aafdab4be9e4d HTTP/1.1" 200 None 2026-03-30 00:43:06,662 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:06,676 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:43:06,678 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.20.0.1:43820 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:43:06,702 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6d63c3aafdab4be9e4d HTTP/1.1" 200 None 2026-03-30 00:43:06,704 - lava_callback - INFO - Completed processing callback for node 69c9c6d63c3aafdab4be9e4d 2026-03-30 00:43:08,908 - lava_callback - INFO - Received callback for node 69c9c6b43c3aafdab4be9dd5 INFO: 172.20.0.1:43828 - "POST /node/69c9c6b43c3aafdab4be9dd5 HTTP/1.0" 202 Accepted 2026-03-30 00:43:08,914 - lava_callback - INFO - Starting async job submit for node 69c9c6b43c3aafdab4be9dd5 2026-03-30 00:43:08,936 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:08,964 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6b43c3aafdab4be9dd5 HTTP/1.1" 200 None 2026-03-30 00:43:08,965 - lava_callback - DEBUG - Processing node 69c9c6b43c3aafdab4be9dd5: baseline-arm-broonie 2026-03-30 00:43:09,200 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:43:09,309 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:43:09,311 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c9c6b43c3aafdab4be9dd5/log.txt.gz 2026-03-30 00:43:09,456 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:43:09,482 - lava_callback - INFO - Received callback for node 69c9c6b83c3aafdab4be9e02 2026-03-30 00:43:09,490 - lava_callback - INFO - Starting async job submit for node 69c9c6b83c3aafdab4be9e02 INFO: 172.20.0.1:43834 - "POST /node/69c9c6b83c3aafdab4be9e02 HTTP/1.0" 202 Accepted 2026-03-30 00:43:09,512 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:09,536 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6b83c3aafdab4be9e02 HTTP/1.1" 200 None 2026-03-30 00:43:09,537 - lava_callback - DEBUG - Processing node 69c9c6b83c3aafdab4be9e02: baseline-arm-clabbe 2026-03-30 00:43:09,719 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:43:09,767 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c9c6b43c3aafdab4be9dd5/lava-logs.yaml 2026-03-30 00:43:09,807 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:43:09,808 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:43:09,951 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:43:09,953 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c9c6b83c3aafdab4be9e02/log.txt.gz 2026-03-30 00:43:09,970 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:43:10,006 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c9c6b43c3aafdab4be9dd5/lava_callback.json.gz 2026-03-30 00:43:10,062 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:10,118 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6b43c3aafdab4be9dd5 HTTP/1.1" 200 None 2026-03-30 00:43:10,121 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:43:10,121 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:10,142 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:43:10,144 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:10,174 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6b43c3aafdab4be9dd5 HTTP/1.1" 200 None 2026-03-30 00:43:10,176 - lava_callback - INFO - Completed processing callback for node 69c9c6b43c3aafdab4be9dd5 2026-03-30 00:43:10,343 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:43:10,345 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c9c6b83c3aafdab4be9e02/lava-logs.yaml 2026-03-30 00:43:10,353 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:43:10,518 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:43:10,520 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c9c6b83c3aafdab4be9e02/lava_callback.json.gz 2026-03-30 00:43:10,544 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:10,567 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6b83c3aafdab4be9e02 HTTP/1.1" 200 None 2026-03-30 00:43:10,568 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:10,590 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:43:10,592 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:10,616 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6b83c3aafdab4be9e02 HTTP/1.1" 200 None 2026-03-30 00:43:10,621 - lava_callback - INFO - Completed processing callback for node 69c9c6b83c3aafdab4be9e02 2026-03-30 00:43:11,137 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:11,163 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 INFO: 172.20.0.1:48422 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48430 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48440 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:43:22,239 - lava_callback - INFO - Received callback for node 69c7242129015d0f9598f8e9 INFO: 172.20.0.1:50738 - "POST /node/69c7242129015d0f9598f8e9 HTTP/1.0" 202 Accepted 2026-03-30 00:43:22,244 - lava_callback - INFO - Starting async job submit for node 69c7242129015d0f9598f8e9 2026-03-30 00:43:22,245 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:22,260 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7242129015d0f9598f8e9 HTTP/1.1" 200 None 2026-03-30 00:43:22,261 - lava_callback - DEBUG - Processing node 69c7242129015d0f9598f8e9: baseline-arm-baylibre 2026-03-30 00:43:22,261 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7242129015d0f9598f8e9 2026-03-30 00:43:22,261 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:22,274 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7242129015d0f9598f8e9 HTTP/1.1" 200 None 2026-03-30 00:43:22,276 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:22,292 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 00:43:22,294 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:22,324 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7242129015d0f9598f8e9 HTTP/1.1" 200 None 2026-03-30 00:43:22,330 - lava_callback - ERROR - Error processing callback for node 69c7242129015d0f9598f8e9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:50754 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35466 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52340 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:43:47,179 - lava_callback - INFO - Received callback for node 69c9c6af3c3aafdab4be9dca INFO: 172.20.0.1:52346 - "POST /node/69c9c6af3c3aafdab4be9dca HTTP/1.0" 202 Accepted 2026-03-30 00:43:47,185 - lava_callback - INFO - Starting async job submit for node 69c9c6af3c3aafdab4be9dca 2026-03-30 00:43:47,207 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:47,222 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6af3c3aafdab4be9dca HTTP/1.1" 200 None 2026-03-30 00:43:47,223 - lava_callback - DEBUG - Processing node 69c9c6af3c3aafdab4be9dca: baseline-arm-broonie 2026-03-30 00:43:47,427 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:43:47,573 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:43:47,574 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c9c6af3c3aafdab4be9dca/log.txt.gz 2026-03-30 00:43:47,696 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:43:47,882 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:43:47,884 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c9c6af3c3aafdab4be9dca/lava-logs.yaml 2026-03-30 00:43:47,891 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:43:48,193 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:43:48,194 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69c9c6af3c3aafdab4be9dca/lava_callback.json.gz 2026-03-30 00:43:48,218 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:48,233 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6af3c3aafdab4be9dca HTTP/1.1" 200 None 2026-03-30 00:43:48,234 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:48,248 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:43:48,249 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:48,274 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6af3c3aafdab4be9dca HTTP/1.1" 200 None 2026-03-30 00:43:48,276 - lava_callback - INFO - Completed processing callback for node 69c9c6af3c3aafdab4be9dca 2026-03-30 00:43:55,392 - lava_callback - INFO - Received callback for node 69c9c6d33c3aafdab4be9e38 INFO: 172.20.0.1:54718 - "POST /node/69c9c6d33c3aafdab4be9e38 HTTP/1.0" 202 Accepted 2026-03-30 00:43:55,398 - lava_callback - INFO - Starting async job submit for node 69c9c6d33c3aafdab4be9e38 2026-03-30 00:43:55,421 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:55,435 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6d33c3aafdab4be9e38 HTTP/1.1" 200 None 2026-03-30 00:43:55,436 - lava_callback - DEBUG - Processing node 69c9c6d33c3aafdab4be9e38: baseline-arm-pengutronix 2026-03-30 00:43:55,591 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:43:55,699 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:43:55,701 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9c6d33c3aafdab4be9e38/log.txt.gz 2026-03-30 00:43:55,791 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:43:55,953 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:43:55,955 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9c6d33c3aafdab4be9e38/lava-logs.yaml 2026-03-30 00:43:55,960 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:43:56,094 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:43:56,096 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9c6d33c3aafdab4be9e38/lava_callback.json.gz 2026-03-30 00:43:56,159 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:56,173 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6d33c3aafdab4be9e38 HTTP/1.1" 200 None 2026-03-30 00:43:56,175 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:56,190 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:43:56,192 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:56,217 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6d33c3aafdab4be9e38 HTTP/1.1" 200 None 2026-03-30 00:43:56,274 - lava_callback - INFO - Completed processing callback for node 69c9c6d33c3aafdab4be9e38 INFO: 172.20.0.1:54726 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:43:56,694 - lava_callback - INFO - Received callback for node 69c9c6d23c3aafdab4be9e35 INFO: 172.20.0.1:54736 - "POST /node/69c9c6d23c3aafdab4be9e35 HTTP/1.0" 202 Accepted 2026-03-30 00:43:56,700 - lava_callback - INFO - Starting async job submit for node 69c9c6d23c3aafdab4be9e35 2026-03-30 00:43:56,731 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:56,744 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6d23c3aafdab4be9e35 HTTP/1.1" 200 None 2026-03-30 00:43:56,745 - lava_callback - DEBUG - Processing node 69c9c6d23c3aafdab4be9e35: baseline-arm-pengutronix 2026-03-30 00:43:56,887 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:43:57,011 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:43:57,013 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9c6d23c3aafdab4be9e35/log.txt.gz 2026-03-30 00:43:57,098 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:43:57,282 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:43:57,284 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9c6d23c3aafdab4be9e35/lava-logs.yaml 2026-03-30 00:43:57,288 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:43:57,424 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:43:57,425 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9c6d23c3aafdab4be9e35/lava_callback.json.gz 2026-03-30 00:43:57,471 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:57,492 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6d23c3aafdab4be9e35 HTTP/1.1" 200 None 2026-03-30 00:43:57,494 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:57,507 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:43:57,509 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:43:57,532 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6d23c3aafdab4be9e35 HTTP/1.1" 200 None 2026-03-30 00:43:57,589 - lava_callback - INFO - Completed processing callback for node 69c9c6d23c3aafdab4be9e35 INFO: 172.20.0.1:41492 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:44:11,166 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:44:11,182 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-30 00:44:14,993 - lava_callback - INFO - Received callback for node 69c9c6ee3c3aafdab4be9e6b INFO: 172.20.0.1:52114 - "POST /node/69c9c6ee3c3aafdab4be9e6b HTTP/1.0" 202 Accepted 2026-03-30 00:44:14,999 - lava_callback - INFO - Starting async job submit for node 69c9c6ee3c3aafdab4be9e6b 2026-03-30 00:44:15,023 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:44:15,037 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6ee3c3aafdab4be9e6b HTTP/1.1" 200 None 2026-03-30 00:44:15,038 - lava_callback - DEBUG - Processing node 69c9c6ee3c3aafdab4be9e6b: ltp-dio INFO: 172.20.0.1:52124 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-30 00:44:15,417 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:44:15,581 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:44:15,583 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-dio-69c9c6ee3c3aafdab4be9e6b/log.txt.gz 2026-03-30 00:44:15,802 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:44:16,001 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:44:16,004 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-dio-69c9c6ee3c3aafdab4be9e6b/lava-logs.yaml 2026-03-30 00:44:16,013 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:44:16,163 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:44:16,165 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-dio-69c9c6ee3c3aafdab4be9e6b/lava_callback.json.gz 2026-03-30 00:44:16,189 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:44:16,204 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6ee3c3aafdab4be9e6b HTTP/1.1" 200 None 2026-03-30 00:44:16,206 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:44:16,220 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:44:16,221 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:44:16,246 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6ee3c3aafdab4be9e6b HTTP/1.1" 200 None 2026-03-30 00:44:16,251 - lava_callback - INFO - Completed processing callback for node 69c9c6ee3c3aafdab4be9e6b INFO: 172.20.0.1:52132 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52144 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34910 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48348 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:44:41,184 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:44:41,201 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-30 00:44:42,321 - lava_callback - INFO - Received callback for node 69c7246d29015d0f9598f8f6 INFO: 172.20.0.1:50212 - "POST /node/69c7246d29015d0f9598f8f6 HTTP/1.0" 202 Accepted 2026-03-30 00:44:42,326 - lava_callback - INFO - Starting async job submit for node 69c7246d29015d0f9598f8f6 2026-03-30 00:44:42,327 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:44:42,342 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7246d29015d0f9598f8f6 HTTP/1.1" 200 None 2026-03-30 00:44:42,342 - lava_callback - DEBUG - Processing node 69c7246d29015d0f9598f8f6: baseline-arm64-baylibre 2026-03-30 00:44:42,343 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7246d29015d0f9598f8f6 2026-03-30 00:44:42,343 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:44:42,356 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7246d29015d0f9598f8f6 HTTP/1.1" 200 None 2026-03-30 00:44:42,358 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:44:42,382 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 00:44:42,383 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:44:42,405 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7246d29015d0f9598f8f6 HTTP/1.1" 200 None 2026-03-30 00:44:42,406 - lava_callback - ERROR - Error processing callback for node 69c7246d29015d0f9598f8f6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:50214 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53952 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:44:56,584 - lava_callback - INFO - Received callback for node 69c9c6b93c3aafdab4be9e0a INFO: 172.20.0.1:53960 - "POST /node/69c9c6b93c3aafdab4be9e0a HTTP/1.0" 202 Accepted 2026-03-30 00:44:56,590 - lava_callback - INFO - Starting async job submit for node 69c9c6b93c3aafdab4be9e0a 2026-03-30 00:44:56,630 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:44:56,645 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6b93c3aafdab4be9e0a HTTP/1.1" 200 None 2026-03-30 00:44:56,645 - lava_callback - DEBUG - Processing node 69c9c6b93c3aafdab4be9e0a: baseline-arm-clabbe 2026-03-30 00:44:56,848 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:44:56,983 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:44:56,985 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c9c6b93c3aafdab4be9e0a/log.txt.gz 2026-03-30 00:44:57,107 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:44:57,297 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:44:57,299 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c9c6b93c3aafdab4be9e0a/lava-logs.yaml 2026-03-30 00:44:57,304 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:44:57,460 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:44:57,462 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c9c6b93c3aafdab4be9e0a/lava_callback.json.gz 2026-03-30 00:44:57,485 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:44:57,501 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6b93c3aafdab4be9e0a HTTP/1.1" 200 None 2026-03-30 00:44:57,502 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:44:57,519 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:44:57,521 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:44:57,553 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6b93c3aafdab4be9e0a HTTP/1.1" 200 None 2026-03-30 00:44:57,555 - lava_callback - INFO - Completed processing callback for node 69c9c6b93c3aafdab4be9e0a INFO: 172.20.0.1:54480 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:45:09,651 - lava_callback - INFO - Received callback for node 69c9c6a03c3aafdab4be9d9c INFO: 172.20.0.1:54482 - "POST /node/69c9c6a03c3aafdab4be9d9c HTTP/1.0" 202 Accepted 2026-03-30 00:45:09,657 - lava_callback - INFO - Starting async job submit for node 69c9c6a03c3aafdab4be9d9c 2026-03-30 00:45:09,680 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:45:09,695 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6a03c3aafdab4be9d9c HTTP/1.1" 200 None 2026-03-30 00:45:09,695 - lava_callback - DEBUG - Processing node 69c9c6a03c3aafdab4be9d9c: baseline-arm 2026-03-30 00:45:09,904 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:45:10,026 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:45:10,027 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c6a03c3aafdab4be9d9c/log.txt.gz 2026-03-30 00:45:10,151 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:45:10,331 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:45:10,333 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c6a03c3aafdab4be9d9c/lava-logs.yaml 2026-03-30 00:45:10,339 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:45:10,499 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:45:10,502 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c6a03c3aafdab4be9d9c/lava_callback.json.gz 2026-03-30 00:45:10,528 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:45:10,541 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6a03c3aafdab4be9d9c HTTP/1.1" 200 None 2026-03-30 00:45:10,542 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:45:10,555 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:45:10,557 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:45:10,580 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6a03c3aafdab4be9d9c HTTP/1.1" 200 None 2026-03-30 00:45:10,587 - lava_callback - INFO - Completed processing callback for node 69c9c6a03c3aafdab4be9d9c 2026-03-30 00:45:11,203 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:45:11,218 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:59714 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59724 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59738 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36628 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39096 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52154 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54290 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:46:02,344 - lava_callback - INFO - Received callback for node 69c724b629015d0f9598f906 INFO: 172.20.0.1:41112 - "POST /node/69c724b629015d0f9598f906 HTTP/1.0" 202 Accepted 2026-03-30 00:46:02,350 - lava_callback - INFO - Starting async job submit for node 69c724b629015d0f9598f906 2026-03-30 00:46:02,351 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:02,383 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c724b629015d0f9598f906 HTTP/1.1" 200 None 2026-03-30 00:46:02,384 - lava_callback - DEBUG - Processing node 69c724b629015d0f9598f906: baseline-arm-baylibre 2026-03-30 00:46:02,384 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c724b629015d0f9598f906 2026-03-30 00:46:02,385 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:02,398 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c724b629015d0f9598f906 HTTP/1.1" 200 None 2026-03-30 00:46:02,399 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:02,436 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 00:46:02,437 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:02,457 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c724b629015d0f9598f906 HTTP/1.1" 200 None 2026-03-30 00:46:02,459 - lava_callback - ERROR - Error processing callback for node 69c724b629015d0f9598f906: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-30 00:46:05,523 - lava_callback - INFO - Received callback for node 69c9c6e73c3aafdab4be9e5b INFO: 172.20.0.1:41118 - "POST /node/69c9c6e73c3aafdab4be9e5b HTTP/1.0" 202 Accepted 2026-03-30 00:46:05,530 - lava_callback - INFO - Starting async job submit for node 69c9c6e73c3aafdab4be9e5b 2026-03-30 00:46:05,554 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:05,568 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6e73c3aafdab4be9e5b HTTP/1.1" 200 None 2026-03-30 00:46:05,569 - lava_callback - DEBUG - Processing node 69c9c6e73c3aafdab4be9e5b: ltp-cap_bounds 2026-03-30 00:46:05,931 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:46:06,219 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:46:06,221 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-cap_bounds-69c9c6e73c3aafdab4be9e5b/log.txt.gz 2026-03-30 00:46:06,428 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.20.0.1:41132 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:46:06,637 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:46:06,639 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-cap_bounds-69c9c6e73c3aafdab4be9e5b/lava-logs.yaml 2026-03-30 00:46:06,647 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:46:06,810 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:46:06,812 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-cap_bounds-69c9c6e73c3aafdab4be9e5b/lava_callback.json.gz 2026-03-30 00:46:06,838 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:06,852 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6e73c3aafdab4be9e5b HTTP/1.1" 200 None 2026-03-30 00:46:06,854 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:06,867 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:46:06,868 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:06,893 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6e73c3aafdab4be9e5b HTTP/1.1" 200 None 2026-03-30 00:46:06,894 - lava_callback - INFO - Completed processing callback for node 69c9c6e73c3aafdab4be9e5b 2026-03-30 00:46:11,220 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:11,246 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:40104 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40116 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:46:16,919 - lava_callback - INFO - Received callback for node 69c9c7c23c3aafdab4be9f30 INFO: 172.20.0.1:40130 - "POST /node/69c9c7c23c3aafdab4be9f30 HTTP/1.0" 202 Accepted 2026-03-30 00:46:16,925 - lava_callback - INFO - Starting async job submit for node 69c9c7c23c3aafdab4be9f30 2026-03-30 00:46:16,928 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:16,942 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c7c23c3aafdab4be9f30 HTTP/1.1" 200 None 2026-03-30 00:46:16,943 - lava_callback - DEBUG - Processing node 69c9c7c23c3aafdab4be9f30: baseline-arm 2026-03-30 00:46:16,948 - lava_callback - WARNING - Failed to upload log 2026-03-30 00:46:16,951 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:46:17,086 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:46:17,087 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c7c23c3aafdab4be9f30/lava-logs.yaml 2026-03-30 00:46:17,089 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:46:17,207 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:46:17,209 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c7c23c3aafdab4be9f30/lava_callback.json.gz 2026-03-30 00:46:17,212 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:17,227 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c7c23c3aafdab4be9f30 HTTP/1.1" 200 None 2026-03-30 00:46:17,228 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:17,242 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:46:17,243 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:17,265 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c7c23c3aafdab4be9f30 HTTP/1.1" 200 None 2026-03-30 00:46:17,270 - lava_callback - INFO - Completed processing callback for node 69c9c7c23c3aafdab4be9f30 INFO: 172.20.0.1:40132 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43418 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:46:28,046 - lava_callback - INFO - Received callback for node 69c9c6e93c3aafdab4be9e5e INFO: 172.20.0.1:43420 - "POST /node/69c9c6e93c3aafdab4be9e5e HTTP/1.0" 202 Accepted 2026-03-30 00:46:28,053 - lava_callback - INFO - Starting async job submit for node 69c9c6e93c3aafdab4be9e5e 2026-03-30 00:46:28,078 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:28,093 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6e93c3aafdab4be9e5e HTTP/1.1" 200 None 2026-03-30 00:46:28,094 - lava_callback - DEBUG - Processing node 69c9c6e93c3aafdab4be9e5e: ltp-containers 2026-03-30 00:46:28,449 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:46:28,582 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:46:28,584 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-containers-69c9c6e93c3aafdab4be9e5e/log.txt.gz 2026-03-30 00:46:28,810 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:46:29,029 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:46:29,031 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-containers-69c9c6e93c3aafdab4be9e5e/lava-logs.yaml 2026-03-30 00:46:29,040 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:46:29,193 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:46:29,195 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-containers-69c9c6e93c3aafdab4be9e5e/lava_callback.json.gz 2026-03-30 00:46:29,221 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:29,238 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6e93c3aafdab4be9e5e HTTP/1.1" 200 None 2026-03-30 00:46:29,240 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:29,256 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:46:29,258 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:29,284 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6e93c3aafdab4be9e5e HTTP/1.1" 200 None 2026-03-30 00:46:29,289 - lava_callback - INFO - Completed processing callback for node 69c9c6e93c3aafdab4be9e5e 2026-03-30 00:46:29,743 - lava_callback - INFO - Received callback for node 69c9c6fb3c3aafdab4be9e8e INFO: 172.20.0.1:56686 - "POST /node/69c9c6fb3c3aafdab4be9e8e HTTP/1.0" 202 Accepted 2026-03-30 00:46:29,755 - lava_callback - INFO - Starting async job submit for node 69c9c6fb3c3aafdab4be9e8e 2026-03-30 00:46:29,786 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:29,805 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6fb3c3aafdab4be9e8e HTTP/1.1" 200 None 2026-03-30 00:46:29,805 - lava_callback - DEBUG - Processing node 69c9c6fb3c3aafdab4be9e8e: ltp-syscalls 2026-03-30 00:46:30,534 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:46:30,674 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:46:30,676 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-syscalls-69c9c6fb3c3aafdab4be9e8e/log.txt.gz 2026-03-30 00:46:31,124 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:46:31,341 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:46:31,343 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-syscalls-69c9c6fb3c3aafdab4be9e8e/lava-logs.yaml 2026-03-30 00:46:31,361 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:46:31,552 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:46:31,554 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-syscalls-69c9c6fb3c3aafdab4be9e8e/lava_callback.json.gz 2026-03-30 00:46:31,580 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:31,595 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6fb3c3aafdab4be9e8e HTTP/1.1" 200 None 2026-03-30 00:46:31,596 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:31,609 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:46:31,611 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:31,638 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6fb3c3aafdab4be9e8e HTTP/1.1" 200 None 2026-03-30 00:46:31,643 - lava_callback - INFO - Completed processing callback for node 69c9c6fb3c3aafdab4be9e8e INFO: 172.20.0.1:56694 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:46:39,706 - lava_callback - INFO - Received callback for node 69c9c6bc3c3aafdab4be9e18 INFO: 172.20.0.1:57384 - "POST /node/69c9c6bc3c3aafdab4be9e18 HTTP/1.0" 202 Accepted 2026-03-30 00:46:39,712 - lava_callback - INFO - Starting async job submit for node 69c9c6bc3c3aafdab4be9e18 2026-03-30 00:46:39,733 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:39,748 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6bc3c3aafdab4be9e18 HTTP/1.1" 200 None 2026-03-30 00:46:39,749 - lava_callback - DEBUG - Processing node 69c9c6bc3c3aafdab4be9e18: baseline-arm-clabbe 2026-03-30 00:46:39,952 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:46:40,066 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:46:40,068 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c9c6bc3c3aafdab4be9e18/log.txt.gz 2026-03-30 00:46:40,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:46:40,441 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:46:40,443 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c9c6bc3c3aafdab4be9e18/lava-logs.yaml 2026-03-30 00:46:40,448 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:46:40,585 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:46:40,587 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c9c6bc3c3aafdab4be9e18/lava_callback.json.gz 2026-03-30 00:46:40,609 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:40,623 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6bc3c3aafdab4be9e18 HTTP/1.1" 200 None 2026-03-30 00:46:40,625 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:40,639 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:46:40,640 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:40,664 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6bc3c3aafdab4be9e18 HTTP/1.1" 200 None 2026-03-30 00:46:40,666 - lava_callback - INFO - Completed processing callback for node 69c9c6bc3c3aafdab4be9e18 2026-03-30 00:46:41,248 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:41,266 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 2026-03-30 00:46:42,481 - lava_callback - INFO - Received callback for node 69c724de29015d0f9598f90e INFO: 172.20.0.1:57388 - "POST /node/69c724de29015d0f9598f90e HTTP/1.0" 202 Accepted 2026-03-30 00:46:42,486 - lava_callback - INFO - Starting async job submit for node 69c724de29015d0f9598f90e 2026-03-30 00:46:42,487 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:42,512 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c724de29015d0f9598f90e HTTP/1.1" 200 None 2026-03-30 00:46:42,513 - lava_callback - DEBUG - Processing node 69c724de29015d0f9598f90e: baseline-arm-baylibre 2026-03-30 00:46:42,513 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c724de29015d0f9598f90e 2026-03-30 00:46:42,514 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:42,527 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c724de29015d0f9598f90e HTTP/1.1" 200 None 2026-03-30 00:46:42,528 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:42,543 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 00:46:42,544 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:46:42,565 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c724de29015d0f9598f90e HTTP/1.1" 200 None 2026-03-30 00:46:42,566 - lava_callback - ERROR - Error processing callback for node 69c724de29015d0f9598f90e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57404 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46892 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:47:04,295 - lava_callback - INFO - Received callback for node 69c9c6ea3c3aafdab4be9e65 INFO: 172.20.0.1:40232 - "POST /node/69c9c6ea3c3aafdab4be9e65 HTTP/1.0" 202 Accepted 2026-03-30 00:47:04,302 - lava_callback - INFO - Starting async job submit for node 69c9c6ea3c3aafdab4be9e65 2026-03-30 00:47:04,326 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:47:04,341 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6ea3c3aafdab4be9e65 HTTP/1.1" 200 None 2026-03-30 00:47:04,342 - lava_callback - DEBUG - Processing node 69c9c6ea3c3aafdab4be9e65: ltp-controllers 2026-03-30 00:47:04,801 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:47:04,948 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:47:04,950 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-controllers-69c9c6ea3c3aafdab4be9e65/log.txt.gz 2026-03-30 00:47:05,204 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:47:05,429 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:47:05,431 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-controllers-69c9c6ea3c3aafdab4be9e65/lava-logs.yaml 2026-03-30 00:47:05,441 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:47:05,588 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:47:05,590 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-controllers-69c9c6ea3c3aafdab4be9e65/lava_callback.json.gz 2026-03-30 00:47:05,615 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:47:05,630 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6ea3c3aafdab4be9e65 HTTP/1.1" 200 None 2026-03-30 00:47:05,631 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:47:05,653 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:47:05,655 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:47:05,681 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6ea3c3aafdab4be9e65 HTTP/1.1" 200 None 2026-03-30 00:47:05,682 - lava_callback - INFO - Completed processing callback for node 69c9c6ea3c3aafdab4be9e65 INFO: 172.20.0.1:40240 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:47:11,269 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:47:11,286 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:33578 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33588 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33600 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:47:22,460 - lava_callback - INFO - Received callback for node 69c7251029015d0f9598f91c INFO: 172.20.0.1:43454 - "POST /node/69c7251029015d0f9598f91c HTTP/1.0" 202 Accepted 2026-03-30 00:47:22,466 - lava_callback - INFO - Starting async job submit for node 69c7251029015d0f9598f91c 2026-03-30 00:47:22,467 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:47:22,480 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7251029015d0f9598f91c HTTP/1.1" 200 None 2026-03-30 00:47:22,481 - lava_callback - DEBUG - Processing node 69c7251029015d0f9598f91c: baseline-arm64-baylibre 2026-03-30 00:47:22,481 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7251029015d0f9598f91c 2026-03-30 00:47:22,481 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:47:22,495 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7251029015d0f9598f91c HTTP/1.1" 200 None 2026-03-30 00:47:22,496 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:47:22,513 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 00:47:22,514 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:47:22,535 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7251029015d0f9598f91c HTTP/1.1" 200 None 2026-03-30 00:47:22,536 - lava_callback - ERROR - Error processing callback for node 69c7251029015d0f9598f91c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43458 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39934 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48568 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58684 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:48:06,061 - lava_callback - INFO - Received callback for node 69c9c6ed3c3aafdab4be9e68 INFO: 172.20.0.1:45948 - "POST /node/69c9c6ed3c3aafdab4be9e68 HTTP/1.0" 202 Accepted 2026-03-30 00:48:06,071 - lava_callback - INFO - Starting async job submit for node 69c9c6ed3c3aafdab4be9e68 2026-03-30 00:48:06,131 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:48:06,172 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6ed3c3aafdab4be9e68 HTTP/1.1" 200 None 2026-03-30 00:48:06,173 - lava_callback - DEBUG - Processing node 69c9c6ed3c3aafdab4be9e68: ltp-cve INFO: 172.20.0.1:45958 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:48:06,747 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:48:07,130 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:48:07,132 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-cve-69c9c6ed3c3aafdab4be9e68/log.txt.gz 2026-03-30 00:48:07,422 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:48:07,830 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:48:07,832 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-cve-69c9c6ed3c3aafdab4be9e68/lava-logs.yaml 2026-03-30 00:48:07,843 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:48:07,987 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:48:07,989 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-cve-69c9c6ed3c3aafdab4be9e68/lava_callback.json.gz 2026-03-30 00:48:08,053 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:48:08,079 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6ed3c3aafdab4be9e68 HTTP/1.1" 200 None 2026-03-30 00:48:08,081 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:48:08,105 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:48:08,107 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:48:08,161 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6ed3c3aafdab4be9e68 HTTP/1.1" 200 None 2026-03-30 00:48:08,163 - lava_callback - INFO - Completed processing callback for node 69c9c6ed3c3aafdab4be9e68 2026-03-30 00:48:11,288 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:48:11,304 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:52482 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52496 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52500 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:48:18,722 - lava_callback - INFO - Received callback for node 69c9c6bd3c3aafdab4be9e1b INFO: 172.20.0.1:52512 - "POST /node/69c9c6bd3c3aafdab4be9e1b HTTP/1.0" 202 Accepted 2026-03-30 00:48:18,728 - lava_callback - INFO - Starting async job submit for node 69c9c6bd3c3aafdab4be9e1b 2026-03-30 00:48:18,754 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:48:18,768 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6bd3c3aafdab4be9e1b HTTP/1.1" 200 None 2026-03-30 00:48:18,769 - lava_callback - DEBUG - Processing node 69c9c6bd3c3aafdab4be9e1b: baseline-arm-clabbe 2026-03-30 00:48:19,137 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:48:19,324 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:48:19,326 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c9c6bd3c3aafdab4be9e1b/log.txt.gz 2026-03-30 00:48:19,458 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:48:19,898 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:48:19,900 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c9c6bd3c3aafdab4be9e1b/lava-logs.yaml 2026-03-30 00:48:19,908 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:48:20,097 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:48:20,099 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69c9c6bd3c3aafdab4be9e1b/lava_callback.json.gz 2026-03-30 00:48:20,121 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:48:20,144 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6bd3c3aafdab4be9e1b HTTP/1.1" 200 None 2026-03-30 00:48:20,145 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:48:20,158 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:48:20,159 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:48:20,183 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6bd3c3aafdab4be9e1b HTTP/1.1" 200 None 2026-03-30 00:48:20,189 - lava_callback - INFO - Completed processing callback for node 69c9c6bd3c3aafdab4be9e1b 2026-03-30 00:48:22,447 - lava_callback - INFO - Received callback for node 69c7255429015d0f9598f927 INFO: 172.20.0.1:46938 - "POST /node/69c7255429015d0f9598f927 HTTP/1.0" 202 Accepted 2026-03-30 00:48:22,454 - lava_callback - INFO - Starting async job submit for node 69c7255429015d0f9598f927 2026-03-30 00:48:22,454 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:48:22,469 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7255429015d0f9598f927 HTTP/1.1" 200 None 2026-03-30 00:48:22,470 - lava_callback - DEBUG - Processing node 69c7255429015d0f9598f927: baseline-arm-baylibre 2026-03-30 00:48:22,470 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7255429015d0f9598f927 2026-03-30 00:48:22,471 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:48:22,484 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7255429015d0f9598f927 HTTP/1.1" 200 None 2026-03-30 00:48:22,485 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:48:22,528 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 00:48:22,529 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:48:22,557 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7255429015d0f9598f927 HTTP/1.1" 200 None 2026-03-30 00:48:22,559 - lava_callback - ERROR - Error processing callback for node 69c7255429015d0f9598f927: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46950 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36000 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:48:41,306 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:48:41,334 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:48588 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36082 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:49:02,551 - lava_callback - INFO - Received callback for node 69c7256a29015d0f9598f955 INFO: 172.20.0.1:47012 - "POST /node/69c7256a29015d0f9598f955 HTTP/1.0" 202 Accepted 2026-03-30 00:49:02,560 - lava_callback - INFO - Starting async job submit for node 69c7256a29015d0f9598f955 2026-03-30 00:49:02,564 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:02,631 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7256a29015d0f9598f955 HTTP/1.1" 200 None 2026-03-30 00:49:02,632 - lava_callback - DEBUG - Processing node 69c7256a29015d0f9598f955: baseline-arm-baylibre 2026-03-30 00:49:02,632 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7256a29015d0f9598f955 2026-03-30 00:49:02,633 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:02,672 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7256a29015d0f9598f955 HTTP/1.1" 200 None 2026-03-30 00:49:02,674 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:02,699 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e3a29015d0f9598f73c HTTP/1.1" 200 None 2026-03-30 00:49:02,701 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:02,733 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7256a29015d0f9598f955 HTTP/1.1" 200 None 2026-03-30 00:49:02,734 - lava_callback - ERROR - Error processing callback for node 69c7256a29015d0f9598f955: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-30 00:49:03,238 - lava_callback - INFO - Received callback for node 69c7256c29015d0f9598f958 INFO: 172.20.0.1:47016 - "POST /node/69c7256c29015d0f9598f958 HTTP/1.0" 202 Accepted 2026-03-30 00:49:03,244 - lava_callback - INFO - Starting async job submit for node 69c7256c29015d0f9598f958 2026-03-30 00:49:03,246 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:03,260 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7256c29015d0f9598f958 HTTP/1.1" 200 None 2026-03-30 00:49:03,261 - lava_callback - DEBUG - Processing node 69c7256c29015d0f9598f958: baseline-arm-baylibre 2026-03-30 00:49:03,261 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7256c29015d0f9598f958 2026-03-30 00:49:03,262 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:03,275 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7256c29015d0f9598f958 HTTP/1.1" 200 None 2026-03-30 00:49:03,277 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:03,291 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e3a29015d0f9598f73c HTTP/1.1" 200 None 2026-03-30 00:49:03,293 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:03,324 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7256c29015d0f9598f958 HTTP/1.1" 200 None 2026-03-30 00:49:03,327 - lava_callback - ERROR - Error processing callback for node 69c7256c29015d0f9598f958: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-30 00:49:03,898 - lava_callback - INFO - Received callback for node 69c7256d29015d0f9598f95d INFO: 172.20.0.1:47028 - "POST /node/69c7256d29015d0f9598f95d HTTP/1.0" 202 Accepted 2026-03-30 00:49:03,907 - lava_callback - INFO - Starting async job submit for node 69c7256d29015d0f9598f95d 2026-03-30 00:49:03,910 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:03,942 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7256d29015d0f9598f95d HTTP/1.1" 200 None 2026-03-30 00:49:03,944 - lava_callback - DEBUG - Processing node 69c7256d29015d0f9598f95d: baseline-arm-baylibre 2026-03-30 00:49:03,944 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7256d29015d0f9598f95d 2026-03-30 00:49:03,945 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:03,970 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7256d29015d0f9598f95d HTTP/1.1" 200 None 2026-03-30 00:49:03,972 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:04,007 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e3a29015d0f9598f73c HTTP/1.1" 200 None 2026-03-30 00:49:04,010 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:04,036 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7256d29015d0f9598f95d HTTP/1.1" 200 None 2026-03-30 00:49:04,037 - lava_callback - ERROR - Error processing callback for node 69c7256d29015d0f9598f95d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-30 00:49:04,585 - lava_callback - INFO - Received callback for node 69c7256f29015d0f9598f960 INFO: 172.20.0.1:47038 - "POST /node/69c7256f29015d0f9598f960 HTTP/1.0" 202 Accepted 2026-03-30 00:49:04,594 - lava_callback - INFO - Starting async job submit for node 69c7256f29015d0f9598f960 2026-03-30 00:49:04,600 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:04,626 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7256f29015d0f9598f960 HTTP/1.1" 200 None 2026-03-30 00:49:04,627 - lava_callback - DEBUG - Processing node 69c7256f29015d0f9598f960: baseline-arm-baylibre 2026-03-30 00:49:04,627 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7256f29015d0f9598f960 2026-03-30 00:49:04,628 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:04,651 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7256f29015d0f9598f960 HTTP/1.1" 200 None 2026-03-30 00:49:04,654 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:04,677 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e3a29015d0f9598f73c HTTP/1.1" 200 None 2026-03-30 00:49:04,680 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:04,719 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7256f29015d0f9598f960 HTTP/1.1" 200 None 2026-03-30 00:49:04,721 - lava_callback - ERROR - Error processing callback for node 69c7256f29015d0f9598f960: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-30 00:49:04,896 - lava_callback - INFO - Received callback for node 69c7257b29015d0f9598f983 INFO: 172.20.0.1:47052 - "POST /node/69c7257b29015d0f9598f983 HTTP/1.0" 202 Accepted 2026-03-30 00:49:04,905 - lava_callback - INFO - Starting async job submit for node 69c7257b29015d0f9598f983 2026-03-30 00:49:04,907 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:04,938 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7257b29015d0f9598f983 HTTP/1.1" 200 None 2026-03-30 00:49:04,940 - lava_callback - DEBUG - Processing node 69c7257b29015d0f9598f983: baseline-arm-clabbe 2026-03-30 00:49:04,940 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7257b29015d0f9598f983 2026-03-30 00:49:04,941 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:04,962 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7257b29015d0f9598f983 HTTP/1.1" 200 None 2026-03-30 00:49:04,964 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:04,982 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e3a29015d0f9598f73c HTTP/1.1" 200 None 2026-03-30 00:49:04,983 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:05,016 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7257b29015d0f9598f983 HTTP/1.1" 200 None 2026-03-30 00:49:05,018 - lava_callback - ERROR - Error processing callback for node 69c7257b29015d0f9598f983: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-30 00:49:05,263 - lava_callback - INFO - Received callback for node 69c7257129015d0f9598f96b INFO: 172.20.0.1:47068 - "POST /node/69c7257129015d0f9598f96b HTTP/1.0" 202 Accepted 2026-03-30 00:49:05,271 - lava_callback - INFO - Starting async job submit for node 69c7257129015d0f9598f96b 2026-03-30 00:49:05,272 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:05,316 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7257129015d0f9598f96b HTTP/1.1" 200 None 2026-03-30 00:49:05,317 - lava_callback - DEBUG - Processing node 69c7257129015d0f9598f96b: baseline-arm-baylibre 2026-03-30 00:49:05,317 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7257129015d0f9598f96b 2026-03-30 00:49:05,318 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:05,337 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7257129015d0f9598f96b HTTP/1.1" 200 None 2026-03-30 00:49:05,339 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:05,369 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e3a29015d0f9598f73c HTTP/1.1" 200 None 2026-03-30 00:49:05,371 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:05,404 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7257129015d0f9598f96b HTTP/1.1" 200 None 2026-03-30 00:49:05,405 - lava_callback - ERROR - Error processing callback for node 69c7257129015d0f9598f96b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-30 00:49:05,930 - lava_callback - INFO - Received callback for node 69c7257329015d0f9598f96e INFO: 172.20.0.1:47078 - "POST /node/69c7257329015d0f9598f96e HTTP/1.0" 202 Accepted 2026-03-30 00:49:05,936 - lava_callback - INFO - Starting async job submit for node 69c7257329015d0f9598f96e 2026-03-30 00:49:05,937 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:05,951 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7257329015d0f9598f96e HTTP/1.1" 200 None 2026-03-30 00:49:05,951 - lava_callback - DEBUG - Processing node 69c7257329015d0f9598f96e: baseline-arm-baylibre 2026-03-30 00:49:05,952 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7257329015d0f9598f96e 2026-03-30 00:49:05,952 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:05,975 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7257329015d0f9598f96e HTTP/1.1" 200 None 2026-03-30 00:49:05,976 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:05,990 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e3a29015d0f9598f73c HTTP/1.1" 200 None 2026-03-30 00:49:05,992 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:06,015 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7257329015d0f9598f96e HTTP/1.1" 200 None 2026-03-30 00:49:06,016 - lava_callback - ERROR - Error processing callback for node 69c7257329015d0f9598f96e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47088 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55676 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55686 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55700 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:49:22,516 - lava_callback - INFO - Received callback for node 69c7257d29015d0f9598f985 INFO: 172.20.0.1:53122 - "POST /node/69c7257d29015d0f9598f985 HTTP/1.0" 202 Accepted 2026-03-30 00:49:22,525 - lava_callback - INFO - Starting async job submit for node 69c7257d29015d0f9598f985 2026-03-30 00:49:22,526 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:22,558 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7257d29015d0f9598f985 HTTP/1.1" 200 None 2026-03-30 00:49:22,559 - lava_callback - DEBUG - Processing node 69c7257d29015d0f9598f985: baseline-arm-baylibre 2026-03-30 00:49:22,560 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7257d29015d0f9598f985 2026-03-30 00:49:22,560 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:22,583 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7257d29015d0f9598f985 HTTP/1.1" 200 None 2026-03-30 00:49:22,585 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:22,609 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 00:49:22,611 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:22,641 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7257d29015d0f9598f985 HTTP/1.1" 200 None 2026-03-30 00:49:22,642 - lava_callback - ERROR - Error processing callback for node 69c7257d29015d0f9598f985: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-30 00:49:24,964 - lava_callback - INFO - Received callback for node 69c7258029015d0f9598f993 INFO: 172.20.0.1:53132 - "POST /node/69c7258029015d0f9598f993 HTTP/1.0" 202 Accepted 2026-03-30 00:49:24,974 - lava_callback - INFO - Starting async job submit for node 69c7258029015d0f9598f993 2026-03-30 00:49:24,974 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:24,996 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7258029015d0f9598f993 HTTP/1.1" 200 None 2026-03-30 00:49:24,999 - lava_callback - DEBUG - Processing node 69c7258029015d0f9598f993: baseline-arm-clabbe 2026-03-30 00:49:24,999 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7258029015d0f9598f993 2026-03-30 00:49:25,000 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:25,015 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7258029015d0f9598f993 HTTP/1.1" 200 None 2026-03-30 00:49:25,018 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:25,041 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e3a29015d0f9598f73c HTTP/1.1" 200 None 2026-03-30 00:49:25,043 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:25,064 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7258029015d0f9598f993 HTTP/1.1" 200 None 2026-03-30 00:49:25,073 - lava_callback - ERROR - Error processing callback for node 69c7258029015d0f9598f993: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-30 00:49:25,650 - lava_callback - INFO - Received callback for node 69c7258129015d0f9598f997 INFO: 172.20.0.1:53142 - "POST /node/69c7258129015d0f9598f997 HTTP/1.0" 202 Accepted 2026-03-30 00:49:25,659 - lava_callback - INFO - Starting async job submit for node 69c7258129015d0f9598f997 2026-03-30 00:49:25,661 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:25,691 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7258129015d0f9598f997 HTTP/1.1" 200 None 2026-03-30 00:49:25,692 - lava_callback - DEBUG - Processing node 69c7258129015d0f9598f997: baseline-arm-clabbe 2026-03-30 00:49:25,693 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7258129015d0f9598f997 2026-03-30 00:49:25,694 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:25,719 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7258129015d0f9598f997 HTTP/1.1" 200 None 2026-03-30 00:49:25,721 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:25,742 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e3a29015d0f9598f73c HTTP/1.1" 200 None 2026-03-30 00:49:25,745 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:25,774 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7258129015d0f9598f997 HTTP/1.1" 200 None 2026-03-30 00:49:25,780 - lava_callback - ERROR - Error processing callback for node 69c7258129015d0f9598f997: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-30 00:49:26,327 - lava_callback - INFO - Received callback for node 69c7258129015d0f9598f99a INFO: 172.20.0.1:53152 - "POST /node/69c7258129015d0f9598f99a HTTP/1.0" 202 Accepted 2026-03-30 00:49:26,336 - lava_callback - INFO - Starting async job submit for node 69c7258129015d0f9598f99a 2026-03-30 00:49:26,337 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:26,369 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7258129015d0f9598f99a HTTP/1.1" 200 None 2026-03-30 00:49:26,371 - lava_callback - DEBUG - Processing node 69c7258129015d0f9598f99a: baseline-arm-clabbe 2026-03-30 00:49:26,371 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7258129015d0f9598f99a 2026-03-30 00:49:26,372 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:26,396 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7258129015d0f9598f99a HTTP/1.1" 200 None 2026-03-30 00:49:26,399 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:26,422 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e3a29015d0f9598f73c HTTP/1.1" 200 None 2026-03-30 00:49:26,424 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:26,460 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7258129015d0f9598f99a HTTP/1.1" 200 None 2026-03-30 00:49:26,462 - lava_callback - ERROR - Error processing callback for node 69c7258129015d0f9598f99a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:53158 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:49:27,032 - lava_callback - INFO - Received callback for node 69c7258429015d0f9598f9a7 INFO: 172.20.0.1:53168 - "POST /node/69c7258429015d0f9598f9a7 HTTP/1.0" 202 Accepted 2026-03-30 00:49:27,042 - lava_callback - INFO - Starting async job submit for node 69c7258429015d0f9598f9a7 2026-03-30 00:49:27,044 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:27,067 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7258429015d0f9598f9a7 HTTP/1.1" 200 None 2026-03-30 00:49:27,068 - lava_callback - DEBUG - Processing node 69c7258429015d0f9598f9a7: baseline-arm-clabbe 2026-03-30 00:49:27,069 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7258429015d0f9598f9a7 2026-03-30 00:49:27,070 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:27,109 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7258429015d0f9598f9a7 HTTP/1.1" 200 None 2026-03-30 00:49:27,111 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:27,135 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e3a29015d0f9598f73c HTTP/1.1" 200 None 2026-03-30 00:49:27,138 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:49:27,165 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7258429015d0f9598f9a7 HTTP/1.1" 200 None 2026-03-30 00:49:27,172 - lava_callback - ERROR - Error processing callback for node 69c7258429015d0f9598f9a7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43456 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44064 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:50:05,049 - lava_callback - INFO - Received callback for node 69c725aa29015d0f9598fa1d INFO: 172.20.0.1:46342 - "POST /node/69c725aa29015d0f9598fa1d HTTP/1.0" 202 Accepted 2026-03-30 00:50:05,057 - lava_callback - INFO - Starting async job submit for node 69c725aa29015d0f9598fa1d 2026-03-30 00:50:05,058 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:50:05,082 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c725aa29015d0f9598fa1d HTTP/1.1" 200 None 2026-03-30 00:50:05,084 - lava_callback - DEBUG - Processing node 69c725aa29015d0f9598fa1d: baseline-arm-clabbe 2026-03-30 00:50:05,084 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c725aa29015d0f9598fa1d 2026-03-30 00:50:05,084 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:50:05,108 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c725aa29015d0f9598fa1d HTTP/1.1" 200 None 2026-03-30 00:50:05,110 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:50:05,133 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 00:50:05,135 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:50:05,171 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c725aa29015d0f9598fa1d HTTP/1.1" 200 None 2026-03-30 00:50:05,173 - lava_callback - ERROR - Error processing callback for node 69c725aa29015d0f9598fa1d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46356 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:50:07,399 - lava_callback - INFO - Received callback for node 69c9c6f03c3aafdab4be9e6e INFO: 172.20.0.1:46372 - "POST /node/69c9c6f03c3aafdab4be9e6e HTTP/1.0" 202 Accepted 2026-03-30 00:50:07,413 - lava_callback - INFO - Starting async job submit for node 69c9c6f03c3aafdab4be9e6e 2026-03-30 00:50:07,460 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:50:07,498 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6f03c3aafdab4be9e6e HTTP/1.1" 200 None 2026-03-30 00:50:07,500 - lava_callback - DEBUG - Processing node 69c9c6f03c3aafdab4be9e6e: ltp-fs_bind 2026-03-30 00:50:08,131 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:50:08,536 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:50:08,538 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-fs_bind-69c9c6f03c3aafdab4be9e6e/log.txt.gz 2026-03-30 00:50:08,933 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:50:09,507 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:50:09,510 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-fs_bind-69c9c6f03c3aafdab4be9e6e/lava-logs.yaml 2026-03-30 00:50:09,521 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:50:09,958 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:50:09,961 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-fs_bind-69c9c6f03c3aafdab4be9e6e/lava_callback.json.gz 2026-03-30 00:50:10,003 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:50:10,028 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6f03c3aafdab4be9e6e HTTP/1.1" 200 None 2026-03-30 00:50:10,030 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:50:10,050 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:50:10,052 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:50:10,095 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6f03c3aafdab4be9e6e HTTP/1.1" 200 None 2026-03-30 00:50:10,097 - lava_callback - INFO - Completed processing callback for node 69c9c6f03c3aafdab4be9e6e 2026-03-30 00:50:11,337 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:50:11,363 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:43632 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43642 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43656 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:50:26,171 - lava_callback - INFO - Received callback for node 69c9c6f23c3aafdab4be9e71 INFO: 172.20.0.1:57832 - "POST /node/69c9c6f23c3aafdab4be9e71 HTTP/1.0" 202 Accepted 2026-03-30 00:50:26,182 - lava_callback - INFO - Starting async job submit for node 69c9c6f23c3aafdab4be9e71 2026-03-30 00:50:26,228 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:50:26,259 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6f23c3aafdab4be9e71 HTTP/1.1" 200 None 2026-03-30 00:50:26,260 - lava_callback - DEBUG - Processing node 69c9c6f23c3aafdab4be9e71: ltp-fsx INFO: 172.20.0.1:57846 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:50:26,934 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:50:27,370 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:50:27,373 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-fsx-69c9c6f23c3aafdab4be9e71/log.txt.gz 2026-03-30 00:50:27,833 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:50:28,489 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:50:28,492 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-fsx-69c9c6f23c3aafdab4be9e71/lava-logs.yaml 2026-03-30 00:50:28,503 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:50:29,049 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:50:29,051 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-fsx-69c9c6f23c3aafdab4be9e71/lava_callback.json.gz 2026-03-30 00:50:29,098 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:50:29,126 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6f23c3aafdab4be9e71 HTTP/1.1" 200 None 2026-03-30 00:50:29,128 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:50:29,154 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:50:29,156 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:50:29,198 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6f23c3aafdab4be9e71 HTTP/1.1" 200 None 2026-03-30 00:50:29,200 - lava_callback - INFO - Completed processing callback for node 69c9c6f23c3aafdab4be9e71 INFO: 172.20.0.1:37368 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:50:41,367 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:50:41,398 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-30 00:50:42,658 - lava_callback - INFO - Received callback for node 69c725cd29015d0f9598fa5f INFO: 172.20.0.1:47096 - "POST /node/69c725cd29015d0f9598fa5f HTTP/1.0" 202 Accepted 2026-03-30 00:50:42,666 - lava_callback - INFO - Starting async job submit for node 69c725cd29015d0f9598fa5f 2026-03-30 00:50:42,673 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:50:42,698 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c725cd29015d0f9598fa5f HTTP/1.1" 200 None 2026-03-30 00:50:42,699 - lava_callback - DEBUG - Processing node 69c725cd29015d0f9598fa5f: baseline-arm-baylibre 2026-03-30 00:50:42,699 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c725cd29015d0f9598fa5f 2026-03-30 00:50:42,700 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:50:42,745 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c725cd29015d0f9598fa5f HTTP/1.1" 200 None 2026-03-30 00:50:42,748 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:50:42,787 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 00:50:42,790 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:50:42,848 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c725cd29015d0f9598fa5f HTTP/1.1" 200 None 2026-03-30 00:50:42,850 - lava_callback - ERROR - Error processing callback for node 69c725cd29015d0f9598fa5f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47100 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43618 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36602 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37550 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37562 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:51:16,761 - lava_callback - INFO - Received callback for node 69c9c4553c3aafdab4be9d1f INFO: 172.20.0.1:37572 - "POST /node/69c9c4553c3aafdab4be9d1f HTTP/1.0" 202 Accepted 2026-03-30 00:51:16,770 - lava_callback - INFO - Starting async job submit for node 69c9c4553c3aafdab4be9d1f 2026-03-30 00:51:16,863 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:51:16,889 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c4553c3aafdab4be9d1f HTTP/1.1" 200 None 2026-03-30 00:51:16,890 - lava_callback - DEBUG - Processing node 69c9c4553c3aafdab4be9d1f: ltp-smoke 2026-03-30 00:51:17,664 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.20.0.1:37578 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:51:18,613 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:51:18,615 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69c9c4553c3aafdab4be9d1f/log.txt.gz 2026-03-30 00:51:18,746 - lava_callback - INFO - Received callback for node 69c9c6f33c3aafdab4be9e7f INFO: 172.20.0.1:37582 - "POST /node/69c9c6f33c3aafdab4be9e7f HTTP/1.0" 202 Accepted 2026-03-30 00:51:18,768 - lava_callback - INFO - Starting async job submit for node 69c9c6f33c3aafdab4be9e7f 2026-03-30 00:51:18,872 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:51:18,940 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6f33c3aafdab4be9e7f HTTP/1.1" 200 None 2026-03-30 00:51:18,947 - lava_callback - DEBUG - Processing node 69c9c6f33c3aafdab4be9e7f: ltp-input 2026-03-30 00:51:19,254 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:51:19,731 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:51:20,445 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:51:20,448 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69c9c4553c3aafdab4be9d1f/lava-logs.yaml 2026-03-30 00:51:20,462 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:51:20,647 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:51:20,649 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-input-69c9c6f33c3aafdab4be9e7f/log.txt.gz 2026-03-30 00:51:21,021 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:51:21,567 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:51:21,570 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69c9c4553c3aafdab4be9d1f/lava_callback.json.gz 2026-03-30 00:51:21,749 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:51:21,771 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c4553c3aafdab4be9d1f HTTP/1.1" 200 None 2026-03-30 00:51:21,774 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:51:21,790 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1213c3aafdab4be774c HTTP/1.1" 200 None 2026-03-30 00:51:21,792 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:51:21,838 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c4553c3aafdab4be9d1f HTTP/1.1" 200 None 2026-03-30 00:51:22,029 - lava_callback - INFO - Completed processing callback for node 69c9c4553c3aafdab4be9d1f 2026-03-30 00:51:22,240 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:51:22,242 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-input-69c9c6f33c3aafdab4be9e7f/lava-logs.yaml 2026-03-30 00:51:22,253 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:51:22,633 - lava_callback - INFO - Received callback for node 69c725f729015d0f9598fa8c INFO: 172.20.0.1:51316 - "POST /node/69c725f729015d0f9598fa8c HTTP/1.0" 202 Accepted 2026-03-30 00:51:22,642 - lava_callback - INFO - Starting async job submit for node 69c725f729015d0f9598fa8c 2026-03-30 00:51:22,643 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:51:22,668 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c725f729015d0f9598fa8c HTTP/1.1" 200 None 2026-03-30 00:51:22,669 - lava_callback - DEBUG - Processing node 69c725f729015d0f9598fa8c: baseline-arm-baylibre 2026-03-30 00:51:22,669 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c725f729015d0f9598fa8c 2026-03-30 00:51:22,670 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:51:22,689 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c725f729015d0f9598fa8c HTTP/1.1" 200 None 2026-03-30 00:51:22,691 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:51:22,714 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 00:51:22,716 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:51:22,772 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c725f729015d0f9598fa8c HTTP/1.1" 200 None 2026-03-30 00:51:22,774 - lava_callback - ERROR - Error processing callback for node 69c725f729015d0f9598fa8c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-30 00:51:23,321 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:51:23,323 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-input-69c9c6f33c3aafdab4be9e7f/lava_callback.json.gz 2026-03-30 00:51:23,366 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:51:23,415 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6f33c3aafdab4be9e7f HTTP/1.1" 200 None 2026-03-30 00:51:23,418 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:51:23,440 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:51:23,443 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:51:23,472 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6f33c3aafdab4be9e7f HTTP/1.1" 200 None 2026-03-30 00:51:23,473 - lava_callback - INFO - Completed processing callback for node 69c9c6f33c3aafdab4be9e7f INFO: 172.20.0.1:51318 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53560 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:51:41,401 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:51:41,421 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:42094 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37774 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33898 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43314 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43326 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43334 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:52:25,922 - lava_callback - INFO - Received callback for node 69c9c6f63c3aafdab4be9e85 INFO: 172.20.0.1:41268 - "POST /node/69c9c6f63c3aafdab4be9e85 HTTP/1.0" 202 Accepted 2026-03-30 00:52:25,931 - lava_callback - INFO - Starting async job submit for node 69c9c6f63c3aafdab4be9e85 2026-03-30 00:52:25,967 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:52:25,985 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6f63c3aafdab4be9e85 HTTP/1.1" 200 None 2026-03-30 00:52:25,986 - lava_callback - DEBUG - Processing node 69c9c6f63c3aafdab4be9e85: ltp-pty INFO: 172.20.0.1:41274 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:52:26,577 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:52:27,092 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:52:27,094 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-pty-69c9c6f63c3aafdab4be9e85/log.txt.gz 2026-03-30 00:52:27,407 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:52:27,960 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:52:27,963 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-pty-69c9c6f63c3aafdab4be9e85/lava-logs.yaml 2026-03-30 00:52:27,974 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:52:28,512 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:52:28,513 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-pty-69c9c6f63c3aafdab4be9e85/lava_callback.json.gz 2026-03-30 00:52:28,544 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:52:28,566 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6f63c3aafdab4be9e85 HTTP/1.1" 200 None 2026-03-30 00:52:28,568 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:52:28,586 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:52:28,589 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:52:28,632 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6f63c3aafdab4be9e85 HTTP/1.1" 200 None 2026-03-30 00:52:28,634 - lava_callback - INFO - Completed processing callback for node 69c9c6f63c3aafdab4be9e85 2026-03-30 00:52:32,393 - lava_callback - INFO - Received callback for node 69c9c90a3c3aafdab4be9ff3 INFO: 172.20.0.1:46662 - "POST /node/69c9c90a3c3aafdab4be9ff3 HTTP/1.0" 202 Accepted 2026-03-30 00:52:32,402 - lava_callback - INFO - Starting async job submit for node 69c9c90a3c3aafdab4be9ff3 2026-03-30 00:52:32,429 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:52:32,457 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c90a3c3aafdab4be9ff3 HTTP/1.1" 200 None 2026-03-30 00:52:32,458 - lava_callback - DEBUG - Processing node 69c9c90a3c3aafdab4be9ff3: baseline-arm-pengutronix 2026-03-30 00:52:32,704 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:52:33,215 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:52:33,217 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9c90a3c3aafdab4be9ff3/log.txt.gz 2026-03-30 00:52:33,327 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:52:33,838 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:52:33,841 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9c90a3c3aafdab4be9ff3/lava-logs.yaml 2026-03-30 00:52:33,847 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:52:34,302 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:52:34,304 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9c90a3c3aafdab4be9ff3/lava_callback.json.gz 2026-03-30 00:52:34,334 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:52:34,355 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c90a3c3aafdab4be9ff3 HTTP/1.1" 200 None 2026-03-30 00:52:34,357 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:52:34,385 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:52:34,387 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:52:34,435 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c90a3c3aafdab4be9ff3 HTTP/1.1" 200 None 2026-03-30 00:52:34,437 - lava_callback - INFO - Completed processing callback for node 69c9c90a3c3aafdab4be9ff3 INFO: 172.20.0.1:46676 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:52:41,423 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:52:41,444 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:39020 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40902 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57768 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:53:12,444 - lava_callback - INFO - Received callback for node 69c9c9323c3aafdab4bea010 INFO: 172.20.0.1:41270 - "POST /node/69c9c9323c3aafdab4bea010 HTTP/1.0" 202 Accepted 2026-03-30 00:53:12,450 - lava_callback - INFO - Starting async job submit for node 69c9c9323c3aafdab4bea010 2026-03-30 00:53:12,468 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:53:12,481 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c9323c3aafdab4bea010 HTTP/1.1" 200 None 2026-03-30 00:53:12,482 - lava_callback - DEBUG - Processing node 69c9c9323c3aafdab4bea010: baseline-arm-pengutronix 2026-03-30 00:53:12,624 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:53:12,748 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:53:12,750 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9c9323c3aafdab4bea010/log.txt.gz 2026-03-30 00:53:12,834 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:53:13,021 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:53:13,023 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9c9323c3aafdab4bea010/lava-logs.yaml 2026-03-30 00:53:13,028 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:53:13,160 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:53:13,162 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9c9323c3aafdab4bea010/lava_callback.json.gz 2026-03-30 00:53:13,181 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:53:13,195 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c9323c3aafdab4bea010 HTTP/1.1" 200 None 2026-03-30 00:53:13,197 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:53:13,215 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:53:13,217 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:53:13,242 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c9323c3aafdab4bea010 HTTP/1.1" 200 None 2026-03-30 00:53:13,244 - lava_callback - INFO - Completed processing callback for node 69c9c9323c3aafdab4bea010 INFO: 172.20.0.1:41284 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41290 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41302 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:53:25,097 - lava_callback - INFO - Received callback for node 69c7267129015d0f9598facf INFO: 172.20.0.1:36196 - "POST /node/69c7267129015d0f9598facf HTTP/1.0" 202 Accepted 2026-03-30 00:53:25,103 - lava_callback - INFO - Starting async job submit for node 69c7267129015d0f9598facf 2026-03-30 00:53:25,104 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:53:25,118 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7267129015d0f9598facf HTTP/1.1" 200 None 2026-03-30 00:53:25,118 - lava_callback - DEBUG - Processing node 69c7267129015d0f9598facf: baseline-arm-clabbe 2026-03-30 00:53:25,119 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7267129015d0f9598facf 2026-03-30 00:53:25,119 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:53:25,132 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7267129015d0f9598facf HTTP/1.1" 200 None 2026-03-30 00:53:25,133 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:53:25,150 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 00:53:25,152 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:53:25,173 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7267129015d0f9598facf HTTP/1.1" 200 None 2026-03-30 00:53:25,174 - lava_callback - ERROR - Error processing callback for node 69c7267129015d0f9598facf: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:36206 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41782 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:53:41,446 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:53:41,462 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:32978 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49112 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:54:05,112 - lava_callback - INFO - Received callback for node 69c7269b29015d0f9598fae3 INFO: 172.20.0.1:35124 - "POST /node/69c7269b29015d0f9598fae3 HTTP/1.0" 202 Accepted 2026-03-30 00:54:05,118 - lava_callback - INFO - Starting async job submit for node 69c7269b29015d0f9598fae3 2026-03-30 00:54:05,119 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:05,142 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7269b29015d0f9598fae3 HTTP/1.1" 200 None 2026-03-30 00:54:05,143 - lava_callback - DEBUG - Processing node 69c7269b29015d0f9598fae3: baseline-arm-clabbe 2026-03-30 00:54:05,143 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7269b29015d0f9598fae3 2026-03-30 00:54:05,143 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:05,159 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7269b29015d0f9598fae3 HTTP/1.1" 200 None 2026-03-30 00:54:05,160 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:05,174 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 00:54:05,176 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:05,196 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7269b29015d0f9598fae3 HTTP/1.1" 200 None 2026-03-30 00:54:05,198 - lava_callback - ERROR - Error processing callback for node 69c7269b29015d0f9598fae3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35132 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:54:10,669 - lava_callback - INFO - Received callback for node 69c9c6f83c3aafdab4be9e88 INFO: 172.20.0.1:52780 - "POST /node/69c9c6f83c3aafdab4be9e88 HTTP/1.0" 202 Accepted 2026-03-30 00:54:10,675 - lava_callback - INFO - Starting async job submit for node 69c9c6f83c3aafdab4be9e88 2026-03-30 00:54:10,699 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:10,714 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6f83c3aafdab4be9e88 HTTP/1.1" 200 None 2026-03-30 00:54:10,715 - lava_callback - DEBUG - Processing node 69c9c6f83c3aafdab4be9e88: ltp-sched 2026-03-30 00:54:11,053 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:54:11,205 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:54:11,206 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-sched-69c9c6f83c3aafdab4be9e88/log.txt.gz 2026-03-30 00:54:11,432 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:54:11,661 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:54:11,663 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-sched-69c9c6f83c3aafdab4be9e88/lava-logs.yaml 2026-03-30 00:54:11,671 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:54:11,848 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:54:11,850 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-sched-69c9c6f83c3aafdab4be9e88/lava_callback.json.gz 2026-03-30 00:54:11,876 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:11,901 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6f83c3aafdab4be9e88 HTTP/1.1" 200 None 2026-03-30 00:54:11,902 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:11,916 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:54:11,918 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:11,943 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6f83c3aafdab4be9e88 HTTP/1.1" 200 None 2026-03-30 00:54:11,945 - lava_callback - INFO - Completed processing callback for node 69c9c6f83c3aafdab4be9e88 INFO: 172.20.0.1:52794 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52798 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52812 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:54:26,160 - lava_callback - INFO - Received callback for node 69c9c6f93c3aafdab4be9e8b INFO: 172.20.0.1:34960 - "POST /node/69c9c6f93c3aafdab4be9e8b HTTP/1.0" 202 Accepted 2026-03-30 00:54:26,166 - lava_callback - INFO - Starting async job submit for node 69c9c6f93c3aafdab4be9e8b 2026-03-30 00:54:26,191 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:26,218 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6f93c3aafdab4be9e8b HTTP/1.1" 200 None 2026-03-30 00:54:26,219 - lava_callback - DEBUG - Processing node 69c9c6f93c3aafdab4be9e8b: ltp-smoke INFO: 172.20.0.1:34976 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:54:26,565 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:54:26,693 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:54:26,694 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69c9c6f93c3aafdab4be9e8b/log.txt.gz 2026-03-30 00:54:26,902 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:54:27,125 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:54:27,127 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69c9c6f93c3aafdab4be9e8b/lava-logs.yaml 2026-03-30 00:54:27,136 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:54:27,300 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:54:27,302 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69c9c6f93c3aafdab4be9e8b/lava_callback.json.gz 2026-03-30 00:54:27,328 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:27,342 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6f93c3aafdab4be9e8b HTTP/1.1" 200 None 2026-03-30 00:54:27,344 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:27,357 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:54:27,358 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:27,385 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6f93c3aafdab4be9e8b HTTP/1.1" 200 None 2026-03-30 00:54:27,386 - lava_callback - INFO - Completed processing callback for node 69c9c6f93c3aafdab4be9e8b INFO: 172.20.0.1:57208 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:54:37,328 - lava_callback - INFO - Received callback for node 69c9c9bf3c3aafdab4bea059 INFO: 172.20.0.1:57218 - "POST /node/69c9c9bf3c3aafdab4bea059 HTTP/1.0" 202 Accepted 2026-03-30 00:54:37,335 - lava_callback - INFO - Starting async job submit for node 69c9c9bf3c3aafdab4bea059 2026-03-30 00:54:37,338 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:37,355 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c9bf3c3aafdab4bea059 HTTP/1.1" 200 None 2026-03-30 00:54:37,356 - lava_callback - DEBUG - Processing node 69c9c9bf3c3aafdab4bea059: baseline-arm 2026-03-30 00:54:37,361 - lava_callback - WARNING - Failed to upload log 2026-03-30 00:54:37,364 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:54:37,479 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:54:37,481 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c9bf3c3aafdab4bea059/lava-logs.yaml 2026-03-30 00:54:37,482 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:54:37,587 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:54:37,589 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c9bf3c3aafdab4bea059/lava_callback.json.gz 2026-03-30 00:54:37,592 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:37,606 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c9bf3c3aafdab4bea059 HTTP/1.1" 200 None 2026-03-30 00:54:37,607 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:37,623 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:54:37,624 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:37,647 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c9bf3c3aafdab4bea059 HTTP/1.1" 200 None 2026-03-30 00:54:37,655 - lava_callback - INFO - Completed processing callback for node 69c9c9bf3c3aafdab4bea059 INFO: 172.20.0.1:59078 - "GET /node/69c641c0b98d4d3c02068594%5Cn HTTP/1.0" 405 Method Not Allowed 2026-03-30 00:54:41,464 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:41,481 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-30 00:54:45,174 - lava_callback - INFO - Received callback for node 69c726c429015d0f9598faf9 INFO: 172.20.0.1:59090 - "POST /node/69c726c429015d0f9598faf9 HTTP/1.0" 202 Accepted 2026-03-30 00:54:45,179 - lava_callback - INFO - Starting async job submit for node 69c726c429015d0f9598faf9 2026-03-30 00:54:45,180 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:45,195 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c726c429015d0f9598faf9 HTTP/1.1" 200 None 2026-03-30 00:54:45,196 - lava_callback - DEBUG - Processing node 69c726c429015d0f9598faf9: baseline-arm-clabbe 2026-03-30 00:54:45,196 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c726c429015d0f9598faf9 2026-03-30 00:54:45,196 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:45,209 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c726c429015d0f9598faf9 HTTP/1.1" 200 None 2026-03-30 00:54:45,211 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:45,223 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 00:54:45,225 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:54:45,245 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c726c429015d0f9598faf9 HTTP/1.1" 200 None 2026-03-30 00:54:45,250 - lava_callback - ERROR - Error processing callback for node 69c726c429015d0f9598faf9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59102 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43568 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42392 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52744 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52752 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52760 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:55:25,186 - lava_callback - INFO - Received callback for node 69c726ed29015d0f9598fb11 INFO: 172.20.0.1:52650 - "POST /node/69c726ed29015d0f9598fb11 HTTP/1.0" 202 Accepted 2026-03-30 00:55:25,194 - lava_callback - INFO - Starting async job submit for node 69c726ed29015d0f9598fb11 2026-03-30 00:55:25,196 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:55:25,217 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c726ed29015d0f9598fb11 HTTP/1.1" 200 None 2026-03-30 00:55:25,219 - lava_callback - DEBUG - Processing node 69c726ed29015d0f9598fb11: baseline-arm-clabbe 2026-03-30 00:55:25,219 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c726ed29015d0f9598fb11 2026-03-30 00:55:25,220 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:55:25,244 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c726ed29015d0f9598fb11 HTTP/1.1" 200 None 2026-03-30 00:55:25,246 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:55:25,278 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 00:55:25,280 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:55:25,310 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c726ed29015d0f9598fb11 HTTP/1.1" 200 None 2026-03-30 00:55:25,311 - lava_callback - ERROR - Error processing callback for node 69c726ed29015d0f9598fb11: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:52654 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42476 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:55:36,957 - lava_callback - INFO - Received callback for node 69c9c6fc3c3aafdab4be9e91 INFO: 172.20.0.1:42488 - "POST /node/69c9c6fc3c3aafdab4be9e91 HTTP/1.0" 202 Accepted 2026-03-30 00:55:36,964 - lava_callback - INFO - Starting async job submit for node 69c9c6fc3c3aafdab4be9e91 2026-03-30 00:55:37,006 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:55:37,021 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6fc3c3aafdab4be9e91 HTTP/1.1" 200 None 2026-03-30 00:55:37,022 - lava_callback - DEBUG - Processing node 69c9c6fc3c3aafdab4be9e91: ltp-syscalls-ipc 2026-03-30 00:55:37,437 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:55:37,594 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:55:37,596 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-syscalls-ipc-69c9c6fc3c3aafdab4be9e91/log.txt.gz 2026-03-30 00:55:37,872 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:55:38,239 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:55:38,241 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-syscalls-ipc-69c9c6fc3c3aafdab4be9e91/lava-logs.yaml 2026-03-30 00:55:38,250 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:55:38,588 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:55:38,590 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-syscalls-ipc-69c9c6fc3c3aafdab4be9e91/lava_callback.json.gz 2026-03-30 00:55:38,615 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:55:38,630 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6fc3c3aafdab4be9e91 HTTP/1.1" 200 None 2026-03-30 00:55:38,632 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:55:38,645 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:55:38,646 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:55:38,673 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6fc3c3aafdab4be9e91 HTTP/1.1" 200 None 2026-03-30 00:55:38,674 - lava_callback - INFO - Completed processing callback for node 69c9c6fc3c3aafdab4be9e91 2026-03-30 00:55:41,483 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:55:41,500 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:39580 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33600 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:56:05,282 - lava_callback - INFO - Received callback for node 69c7271629015d0f959920be INFO: 172.20.0.1:46740 - "POST /node/69c7271629015d0f959920be HTTP/1.0" 202 Accepted 2026-03-30 00:56:05,288 - lava_callback - INFO - Starting async job submit for node 69c7271629015d0f959920be 2026-03-30 00:56:05,289 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:56:05,305 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7271629015d0f959920be HTTP/1.1" 200 None 2026-03-30 00:56:05,306 - lava_callback - DEBUG - Processing node 69c7271629015d0f959920be: baseline-arm-clabbe 2026-03-30 00:56:05,306 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7271629015d0f959920be 2026-03-30 00:56:05,307 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:56:05,323 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7271629015d0f959920be HTTP/1.1" 200 None 2026-03-30 00:56:05,324 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:56:05,338 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 00:56:05,339 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:56:05,361 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7271629015d0f959920be HTTP/1.1" 200 None 2026-03-30 00:56:05,367 - lava_callback - ERROR - Error processing callback for node 69c7271629015d0f959920be: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46744 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55342 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55352 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55354 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39178 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44576 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:56:46,044 - lava_callback - INFO - Received callback for node 69c9c6fe3c3aafdab4be9e94 INFO: 172.20.0.1:59974 - "POST /node/69c9c6fe3c3aafdab4be9e94 HTTP/1.0" 202 Accepted 2026-03-30 00:56:46,051 - lava_callback - INFO - Starting async job submit for node 69c9c6fe3c3aafdab4be9e94 2026-03-30 00:56:46,092 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:56:46,114 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6fe3c3aafdab4be9e94 HTTP/1.1" 200 None 2026-03-30 00:56:46,115 - lava_callback - DEBUG - Processing node 69c9c6fe3c3aafdab4be9e94: ltp-timers INFO: 172.20.0.1:59986 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:56:46,619 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:56:46,863 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:56:46,865 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-timers-69c9c6fe3c3aafdab4be9e94/log.txt.gz 2026-03-30 00:56:47,121 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:56:47,494 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:56:47,497 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-timers-69c9c6fe3c3aafdab4be9e94/lava-logs.yaml 2026-03-30 00:56:47,507 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:56:47,706 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:56:47,708 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-timers-69c9c6fe3c3aafdab4be9e94/lava_callback.json.gz 2026-03-30 00:56:47,743 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:56:47,758 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6fe3c3aafdab4be9e94 HTTP/1.1" 200 None 2026-03-30 00:56:47,760 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:56:47,774 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:56:47,776 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:56:47,816 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6fe3c3aafdab4be9e94 HTTP/1.1" 200 None 2026-03-30 00:56:47,817 - lava_callback - INFO - Completed processing callback for node 69c9c6fe3c3aafdab4be9e94 INFO: 172.20.0.1:37320 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43132 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:57:11,503 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:57:11,526 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:51320 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51334 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51344 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:57:23,145 - lava_callback - INFO - Received callback for node 69c7276d29015d0f959920e1 INFO: 172.20.0.1:38282 - "POST /node/69c7276d29015d0f959920e1 HTTP/1.0" 202 Accepted 2026-03-30 00:57:23,151 - lava_callback - INFO - Starting async job submit for node 69c7276d29015d0f959920e1 2026-03-30 00:57:23,152 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:57:23,166 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7276d29015d0f959920e1 HTTP/1.1" 200 None 2026-03-30 00:57:23,167 - lava_callback - DEBUG - Processing node 69c7276d29015d0f959920e1: baseline-arm64-baylibre 2026-03-30 00:57:23,167 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7276d29015d0f959920e1 2026-03-30 00:57:23,168 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:57:23,181 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7276d29015d0f959920e1 HTTP/1.1" 200 None 2026-03-30 00:57:23,182 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:57:23,196 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 00:57:23,197 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:57:23,225 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7276d29015d0f959920e1 HTTP/1.1" 200 None 2026-03-30 00:57:23,230 - lava_callback - ERROR - Error processing callback for node 69c7276d29015d0f959920e1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38298 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50586 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49360 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40396 - "GET /node/6990ace2c45febfe44521aa4%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:40398 - "GET /node/693e19ef407f32424eb56dab%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:40412 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:58:05,873 - lava_callback - INFO - Received callback for node 69c9c6ff3c3aafdab4be9e97 INFO: 172.20.0.1:54282 - "POST /node/69c9c6ff3c3aafdab4be9e97 HTTP/1.0" 202 Accepted 2026-03-30 00:58:05,879 - lava_callback - INFO - Starting async job submit for node 69c9c6ff3c3aafdab4be9e97 2026-03-30 00:58:05,905 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:58:05,921 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6ff3c3aafdab4be9e97 HTTP/1.1" 200 None 2026-03-30 00:58:05,922 - lava_callback - DEBUG - Processing node 69c9c6ff3c3aafdab4be9e97: ltp-watchqueue 2026-03-30 00:58:06,312 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:58:06,485 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:58:06,487 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-watchqueue-69c9c6ff3c3aafdab4be9e97/log.txt.gz INFO: 172.20.0.1:54288 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:58:06,739 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:58:07,000 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:58:07,002 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-watchqueue-69c9c6ff3c3aafdab4be9e97/lava-logs.yaml 2026-03-30 00:58:07,011 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:58:07,192 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:58:07,194 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-watchqueue-69c9c6ff3c3aafdab4be9e97/lava_callback.json.gz 2026-03-30 00:58:07,231 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:58:07,257 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6ff3c3aafdab4be9e97 HTTP/1.1" 200 None 2026-03-30 00:58:07,259 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:58:07,298 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:58:07,300 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:58:07,325 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6ff3c3aafdab4be9e97 HTTP/1.1" 200 None 2026-03-30 00:58:07,327 - lava_callback - INFO - Completed processing callback for node 69c9c6ff3c3aafdab4be9e97 2026-03-30 00:58:11,528 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:58:11,552 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:44606 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44612 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44614 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60666 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:58:35,036 - lava_callback - INFO - Received callback for node 69c9c6a23c3aafdab4be9da5 INFO: 172.20.0.1:49014 - "POST /node/69c9c6a23c3aafdab4be9da5 HTTP/1.0" 202 Accepted 2026-03-30 00:58:35,043 - lava_callback - INFO - Starting async job submit for node 69c9c6a23c3aafdab4be9da5 2026-03-30 00:58:35,095 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:58:35,110 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6a23c3aafdab4be9da5 HTTP/1.1" 200 None 2026-03-30 00:58:35,111 - lava_callback - DEBUG - Processing node 69c9c6a23c3aafdab4be9da5: baseline-arm 2026-03-30 00:58:35,710 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:58:35,852 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:58:35,853 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c6a23c3aafdab4be9da5/log.txt.gz 2026-03-30 00:58:36,205 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 00:58:36,456 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:58:36,458 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c6a23c3aafdab4be9da5/lava-logs.yaml 2026-03-30 00:58:36,473 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.20.0.1:49026 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:58:36,648 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 00:58:36,650 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c9c6a23c3aafdab4be9da5/lava_callback.json.gz DEBUG: Setup failed for 69c9c6a23c3aafdab4be9da5. Transitting job node result: incomplete -> fail 2026-03-30 00:58:36,759 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:58:36,774 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6a23c3aafdab4be9da5 HTTP/1.1" 200 None 2026-03-30 00:58:36,775 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:58:36,788 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 00:58:36,789 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:58:36,814 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6a23c3aafdab4be9da5 HTTP/1.1" 200 None 2026-03-30 00:58:36,925 - lava_callback - INFO - Completed processing callback for node 69c9c6a23c3aafdab4be9da5 2026-03-30 00:58:41,554 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:58:41,595 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-30 00:58:45,382 - lava_callback - INFO - Received callback for node 69c727b729015d0f959920f7 INFO: 172.20.0.1:50684 - "POST /node/69c727b729015d0f959920f7 HTTP/1.0" 202 Accepted 2026-03-30 00:58:45,387 - lava_callback - INFO - Starting async job submit for node 69c727b729015d0f959920f7 2026-03-30 00:58:45,388 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:58:45,403 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c727b729015d0f959920f7 HTTP/1.1" 200 None 2026-03-30 00:58:45,404 - lava_callback - DEBUG - Processing node 69c727b729015d0f959920f7: baseline-arm-clabbe 2026-03-30 00:58:45,404 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c727b729015d0f959920f7 2026-03-30 00:58:45,405 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:58:45,418 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c727b729015d0f959920f7 HTTP/1.1" 200 None 2026-03-30 00:58:45,420 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:58:45,433 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 00:58:45,435 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:58:45,464 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c727b729015d0f959920f7 HTTP/1.1" 200 None 2026-03-30 00:58:45,465 - lava_callback - ERROR - Error processing callback for node 69c727b729015d0f959920f7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:50694 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33784 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56752 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43250 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43258 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43264 - "GET / HTTP/1.0" 200 OK 2026-03-30 00:59:25,331 - lava_callback - INFO - Received callback for node 69c727df29015d0f95992111 INFO: 172.20.0.1:35560 - "POST /node/69c727df29015d0f95992111 HTTP/1.0" 202 Accepted 2026-03-30 00:59:25,336 - lava_callback - INFO - Starting async job submit for node 69c727df29015d0f95992111 2026-03-30 00:59:25,337 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:59:25,352 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c727df29015d0f95992111 HTTP/1.1" 200 None 2026-03-30 00:59:25,353 - lava_callback - DEBUG - Processing node 69c727df29015d0f95992111: baseline-arm-clabbe 2026-03-30 00:59:25,353 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c727df29015d0f95992111 2026-03-30 00:59:25,354 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:59:25,367 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c727df29015d0f95992111 HTTP/1.1" 200 None 2026-03-30 00:59:25,368 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:59:25,393 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 00:59:25,395 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 00:59:25,415 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c727df29015d0f95992111 HTTP/1.1" 200 None 2026-03-30 00:59:25,424 - lava_callback - ERROR - Error processing callback for node 69c727df29015d0f95992111: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35566 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42902 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42128 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48526 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:00:05,597 - lava_callback - INFO - Received callback for node 69c7280629015d0f95992125 INFO: 172.20.0.1:44466 - "POST /node/69c7280629015d0f95992125 HTTP/1.0" 202 Accepted 2026-03-30 01:00:05,602 - lava_callback - INFO - Starting async job submit for node 69c7280629015d0f95992125 2026-03-30 01:00:05,603 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:00:05,618 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7280629015d0f95992125 HTTP/1.1" 200 None 2026-03-30 01:00:05,618 - lava_callback - DEBUG - Processing node 69c7280629015d0f95992125: baseline-arm-clabbe 2026-03-30 01:00:05,619 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7280629015d0f95992125 2026-03-30 01:00:05,620 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:00:05,640 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7280629015d0f95992125 HTTP/1.1" 200 None 2026-03-30 01:00:05,642 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:00:05,657 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 01:00:05,658 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:00:05,681 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7280629015d0f95992125 HTTP/1.1" 200 None 2026-03-30 01:00:05,689 - lava_callback - ERROR - Error processing callback for node 69c7280629015d0f95992125: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44474 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44476 - "GET /node/693f7fbe0d8ce4018da964a5%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:36778 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36786 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36790 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33922 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54964 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52348 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51766 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40048 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47020 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47034 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47050 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:01:21,041 - lava_callback - INFO - Received callback for node 69c9c6d53c3aafdab4be9e4a INFO: 172.20.0.1:51194 - "POST /node/69c9c6d53c3aafdab4be9e4a HTTP/1.0" 202 Accepted 2026-03-30 01:01:21,089 - lava_callback - INFO - Starting async job submit for node 69c9c6d53c3aafdab4be9e4a 2026-03-30 01:01:21,113 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:01:21,127 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6d53c3aafdab4be9e4a HTTP/1.1" 200 None 2026-03-30 01:01:21,128 - lava_callback - DEBUG - Processing node 69c9c6d53c3aafdab4be9e4a: baseline-arm-pengutronix 2026-03-30 01:01:21,233 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:01:21,393 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:01:21,395 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9c6d53c3aafdab4be9e4a/log.txt.gz 2026-03-30 01:01:21,454 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:01:21,650 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:01:21,652 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9c6d53c3aafdab4be9e4a/lava-logs.yaml 2026-03-30 01:01:21,655 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:01:21,793 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:01:21,795 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9c6d53c3aafdab4be9e4a/lava_callback.json.gz DEBUG: Setup failed for 69c9c6d53c3aafdab4be9e4a. Transitting job node result: incomplete -> fail 2026-03-30 01:01:21,843 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:01:21,858 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6d53c3aafdab4be9e4a HTTP/1.1" 200 None 2026-03-30 01:01:21,859 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:01:21,878 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 01:01:21,879 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:01:21,902 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6d53c3aafdab4be9e4a HTTP/1.1" 200 None 2026-03-30 01:01:21,965 - lava_callback - INFO - Completed processing callback for node 69c9c6d53c3aafdab4be9e4a INFO: 172.20.0.1:51200 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60988 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:01:41,597 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:01:41,618 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-30 01:01:45,476 - lava_callback - INFO - Received callback for node 69c7287329015d0f9599213b INFO: 172.20.0.1:54946 - "POST /node/69c7287329015d0f9599213b HTTP/1.0" 202 Accepted 2026-03-30 01:01:45,484 - lava_callback - INFO - Starting async job submit for node 69c7287329015d0f9599213b 2026-03-30 01:01:45,485 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:01:45,498 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7287329015d0f9599213b HTTP/1.1" 200 None 2026-03-30 01:01:45,499 - lava_callback - DEBUG - Processing node 69c7287329015d0f9599213b: baseline-arm-clabbe 2026-03-30 01:01:45,499 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7287329015d0f9599213b 2026-03-30 01:01:45,499 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:01:45,514 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7287329015d0f9599213b HTTP/1.1" 200 None 2026-03-30 01:01:45,516 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:01:45,536 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 01:01:45,538 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:01:45,558 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7287329015d0f9599213b HTTP/1.1" 200 None 2026-03-30 01:01:45,564 - lava_callback - ERROR - Error processing callback for node 69c7287329015d0f9599213b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48816 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57588 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55646 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55654 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55662 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42546 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42560 - "GET /node/6910152cf4a91d81a0ec1e0b%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:58460 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:02:45,487 - lava_callback - INFO - Received callback for node 69c728ab29015d0f95992151 INFO: 172.20.0.1:57918 - "POST /node/69c728ab29015d0f95992151 HTTP/1.0" 202 Accepted 2026-03-30 01:02:45,493 - lava_callback - INFO - Starting async job submit for node 69c728ab29015d0f95992151 2026-03-30 01:02:45,494 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:02:45,514 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c728ab29015d0f95992151 HTTP/1.1" 200 None 2026-03-30 01:02:45,515 - lava_callback - DEBUG - Processing node 69c728ab29015d0f95992151: baseline-arm64-clabbe 2026-03-30 01:02:45,515 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c728ab29015d0f95992151 2026-03-30 01:02:45,516 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:02:45,532 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c728ab29015d0f95992151 HTTP/1.1" 200 None 2026-03-30 01:02:45,533 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:02:45,555 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 01:02:45,557 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:02:45,578 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c728ab29015d0f95992151 HTTP/1.1" 200 None 2026-03-30 01:02:45,583 - lava_callback - ERROR - Error processing callback for node 69c728ab29015d0f95992151: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57932 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44862 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52222 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48616 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48622 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48636 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52712 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43540 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41774 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:04:05,664 - lava_callback - INFO - Received callback for node 69c728fd29015d0f95992163 INFO: 172.20.0.1:39528 - "POST /node/69c728fd29015d0f95992163 HTTP/1.0" 202 Accepted 2026-03-30 01:04:05,671 - lava_callback - INFO - Starting async job submit for node 69c728fd29015d0f95992163 2026-03-30 01:04:05,672 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:04:05,694 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c728fd29015d0f95992163 HTTP/1.1" 200 None 2026-03-30 01:04:05,695 - lava_callback - DEBUG - Processing node 69c728fd29015d0f95992163: baseline-arm64-clabbe 2026-03-30 01:04:05,695 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c728fd29015d0f95992163 2026-03-30 01:04:05,696 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:04:05,709 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c728fd29015d0f95992163 HTTP/1.1" 200 None 2026-03-30 01:04:05,710 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:04:05,723 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 01:04:05,724 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:04:05,756 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c728fd29015d0f95992163 HTTP/1.1" 200 None 2026-03-30 01:04:05,758 - lava_callback - ERROR - Error processing callback for node 69c728fd29015d0f95992163: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:39532 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59172 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59184 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59200 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37116 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39094 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51576 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51174 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44704 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44412 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44418 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44422 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:05:25,620 - lava_callback - INFO - Received callback for node 69c7295329015d0f95992175 INFO: 172.20.0.1:43792 - "POST /node/69c7295329015d0f95992175 HTTP/1.0" 202 Accepted 2026-03-30 01:05:25,626 - lava_callback - INFO - Starting async job submit for node 69c7295329015d0f95992175 2026-03-30 01:05:25,626 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:05:25,644 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7295329015d0f95992175 HTTP/1.1" 200 None 2026-03-30 01:05:25,645 - lava_callback - DEBUG - Processing node 69c7295329015d0f95992175: baseline-arm64-clabbe 2026-03-30 01:05:25,645 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7295329015d0f95992175 2026-03-30 01:05:25,646 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:05:25,669 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7295329015d0f95992175 HTTP/1.1" 200 None 2026-03-30 01:05:25,670 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:05:25,684 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 01:05:25,685 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:05:25,732 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7295329015d0f95992175 HTTP/1.1" 200 None 2026-03-30 01:05:25,733 - lava_callback - ERROR - Error processing callback for node 69c7295329015d0f95992175: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43802 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54336 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54978 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43092 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44900 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50342 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50346 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50358 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48690 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51642 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35394 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51502 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54868 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38680 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38682 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38684 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38688 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45604 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51750 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33596 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34450 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34456 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51276 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33928 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58094 - "GET /node/69a1599d2b7cf147caaf0444%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:58106 - "GET /node/699036b6985e4efca431cc38%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:58118 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59046 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:09:06,145 - lava_callback - INFO - Received callback for node 69c9c6f43c3aafdab4be9e82 INFO: 172.20.0.1:57758 - "POST /node/69c9c6f43c3aafdab4be9e82 HTTP/1.0" 202 Accepted 2026-03-30 01:09:06,193 - lava_callback - INFO - Starting async job submit for node 69c9c6f43c3aafdab4be9e82 2026-03-30 01:09:06,218 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:09:06,232 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6f43c3aafdab4be9e82 HTTP/1.1" 200 None 2026-03-30 01:09:06,233 - lava_callback - DEBUG - Processing node 69c9c6f43c3aafdab4be9e82: ltp-ipc 2026-03-30 01:09:06,346 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:09:06,504 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:09:06,506 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-ipc-69c9c6f43c3aafdab4be9e82/log.txt.gz INFO: 172.20.0.1:57762 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:09:06,559 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:09:06,743 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:09:06,745 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-ipc-69c9c6f43c3aafdab4be9e82/lava-logs.yaml 2026-03-30 01:09:06,748 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:09:06,881 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:09:06,883 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-ipc-69c9c6f43c3aafdab4be9e82/lava_callback.json.gz 2026-03-30 01:09:06,908 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:09:06,923 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c6f43c3aafdab4be9e82 HTTP/1.1" 200 None 2026-03-30 01:09:06,925 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:09:06,938 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 01:09:06,939 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:09:06,966 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9c6f43c3aafdab4be9e82 HTTP/1.1" 200 None 2026-03-30 01:09:07,025 - lava_callback - INFO - Completed processing callback for node 69c9c6f43c3aafdab4be9e82 2026-03-30 01:09:11,621 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:09:11,637 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:46428 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46436 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46452 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53990 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55152 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54588 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48936 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48418 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60366 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60376 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60388 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34146 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33548 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:10:43,862 - lava_callback - INFO - Received callback for node 69c9cd733c3aafdab4bea1b9 INFO: 172.20.0.1:57158 - "POST /node/69c9cd733c3aafdab4bea1b9 HTTP/1.0" 202 Accepted 2026-03-30 01:10:43,868 - lava_callback - INFO - Starting async job submit for node 69c9cd733c3aafdab4bea1b9 2026-03-30 01:10:43,871 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:10:43,886 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cd733c3aafdab4bea1b9 HTTP/1.1" 200 None 2026-03-30 01:10:43,887 - lava_callback - DEBUG - Processing node 69c9cd733c3aafdab4bea1b9: baseline-arm 2026-03-30 01:10:43,892 - lava_callback - WARNING - Failed to upload log 2026-03-30 01:10:43,895 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:10:44,055 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:10:44,057 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c9cd733c3aafdab4bea1b9/lava-logs.yaml 2026-03-30 01:10:44,058 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:10:44,348 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:10:44,350 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c9cd733c3aafdab4bea1b9/lava_callback.json.gz 2026-03-30 01:10:44,353 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:10:44,367 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cd733c3aafdab4bea1b9 HTTP/1.1" 200 None 2026-03-30 01:10:44,369 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:10:44,383 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 01:10:44,384 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:10:44,406 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9cd733c3aafdab4bea1b9 HTTP/1.1" 200 None 2026-03-30 01:10:44,411 - lava_callback - INFO - Completed processing callback for node 69c9cd733c3aafdab4bea1b9 2026-03-30 01:10:45,820 - lava_callback - INFO - Received callback for node 69c72a8b29015d0f959921fd INFO: 172.20.0.1:57172 - "POST /node/69c72a8b29015d0f959921fd HTTP/1.0" 202 Accepted 2026-03-30 01:10:45,829 - lava_callback - INFO - Starting async job submit for node 69c72a8b29015d0f959921fd 2026-03-30 01:10:45,830 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:10:45,847 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72a8b29015d0f959921fd HTTP/1.1" 200 None 2026-03-30 01:10:45,848 - lava_callback - DEBUG - Processing node 69c72a8b29015d0f959921fd: baseline-arm64-clabbe 2026-03-30 01:10:45,848 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72a8b29015d0f959921fd 2026-03-30 01:10:45,849 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:10:45,863 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72a8b29015d0f959921fd HTTP/1.1" 200 None 2026-03-30 01:10:45,864 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:10:45,879 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 01:10:45,880 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:10:45,904 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72a8b29015d0f959921fd HTTP/1.1" 200 None 2026-03-30 01:10:45,905 - lava_callback - ERROR - Error processing callback for node 69c72a8b29015d0f959921fd: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57184 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36870 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51238 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:11:11,639 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:11:11,656 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:58362 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58370 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58372 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42670 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39060 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53058 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39544 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59266 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49368 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49374 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49378 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44872 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44604 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:12:45,891 - lava_callback - INFO - Received callback for node 69c72b0b29015d0f9599221c INFO: 172.20.0.1:41348 - "POST /node/69c72b0b29015d0f9599221c HTTP/1.0" 202 Accepted 2026-03-30 01:12:45,897 - lava_callback - INFO - Starting async job submit for node 69c72b0b29015d0f9599221c 2026-03-30 01:12:45,898 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:12:45,918 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72b0b29015d0f9599221c HTTP/1.1" 200 None 2026-03-30 01:12:45,920 - lava_callback - DEBUG - Processing node 69c72b0b29015d0f9599221c: baseline-arm64-clabbe 2026-03-30 01:12:45,920 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72b0b29015d0f9599221c 2026-03-30 01:12:45,920 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:12:45,937 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72b0b29015d0f9599221c HTTP/1.1" 200 None 2026-03-30 01:12:45,939 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:12:45,955 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 01:12:45,957 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:12:45,981 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72b0b29015d0f9599221c HTTP/1.1" 200 None 2026-03-30 01:12:45,996 - lava_callback - ERROR - Error processing callback for node 69c72b0b29015d0f9599221c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41364 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35942 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38922 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46036 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46052 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46056 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39358 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42064 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47416 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56140 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55018 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47262 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47276 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47278 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:14:25,947 - lava_callback - INFO - Received callback for node 69c72b6029015d0f95992225 INFO: 172.20.0.1:42946 - "POST /node/69c72b6029015d0f95992225 HTTP/1.0" 202 Accepted 2026-03-30 01:14:25,957 - lava_callback - INFO - Starting async job submit for node 69c72b6029015d0f95992225 2026-03-30 01:14:25,959 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:14:25,999 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72b6029015d0f95992225 HTTP/1.1" 200 None 2026-03-30 01:14:26,001 - lava_callback - DEBUG - Processing node 69c72b6029015d0f95992225: baseline-arm64-clabbe 2026-03-30 01:14:26,001 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72b6029015d0f95992225 2026-03-30 01:14:26,002 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:14:26,035 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72b6029015d0f95992225 HTTP/1.1" 200 None 2026-03-30 01:14:26,038 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:14:26,078 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1679b2b2b51e9ad977955 HTTP/1.1" 200 None 2026-03-30 01:14:26,081 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:14:26,116 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72b6029015d0f95992225 HTTP/1.1" 200 None 2026-03-30 01:14:26,117 - lava_callback - ERROR - Error processing callback for node 69c72b6029015d0f95992225: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:42956 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37534 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53564 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37334 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37340 - "GET /node/693f83b90d8ce4018da96ab4%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:37350 - "GET /node/6985c57c71989fb14149841e%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:48570 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:15:09,669 - lava_callback - INFO - Received callback for node 69c9ce873c3aafdab4bea23a INFO: 172.20.0.1:48580 - "POST /node/69c9ce873c3aafdab4bea23a HTTP/1.0" 202 Accepted 2026-03-30 01:15:09,678 - lava_callback - INFO - Starting async job submit for node 69c9ce873c3aafdab4bea23a 2026-03-30 01:15:09,696 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:09,711 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce873c3aafdab4bea23a HTTP/1.1" 200 None 2026-03-30 01:15:09,712 - lava_callback - DEBUG - Processing node 69c9ce873c3aafdab4bea23a: baseline-arm64 2026-03-30 01:15:09,991 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:15:10,307 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:15:10,309 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce873c3aafdab4bea23a/log.txt.gz 2026-03-30 01:15:10,399 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:15:10,589 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:15:10,591 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce873c3aafdab4bea23a/lava-logs.yaml 2026-03-30 01:15:10,597 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:15:10,783 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:15:10,785 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce873c3aafdab4bea23a/lava_callback.json.gz 2026-03-30 01:15:10,802 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:10,817 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce873c3aafdab4bea23a HTTP/1.1" 200 None 2026-03-30 01:15:10,819 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:10,832 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:15:10,833 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:10,857 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce873c3aafdab4bea23a HTTP/1.1" 200 None 2026-03-30 01:15:10,865 - lava_callback - INFO - Completed processing callback for node 69c9ce873c3aafdab4bea23a 2026-03-30 01:15:11,659 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:11,688 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:41462 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-30 01:15:15,325 - lava_callback - INFO - Received callback for node 69c9ce7a3c3aafdab4bea219 INFO: 172.20.0.1:41466 - "POST /node/69c9ce7a3c3aafdab4bea219 HTTP/1.0" 202 Accepted 2026-03-30 01:15:15,332 - lava_callback - INFO - Starting async job submit for node 69c9ce7a3c3aafdab4bea219 2026-03-30 01:15:15,352 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:15,393 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce7a3c3aafdab4bea219 HTTP/1.1" 200 None 2026-03-30 01:15:15,394 - lava_callback - DEBUG - Processing node 69c9ce7a3c3aafdab4bea219: baseline-arm64 2026-03-30 01:15:15,637 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:15:15,801 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:15:15,803 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce7a3c3aafdab4bea219/log.txt.gz 2026-03-30 01:15:15,904 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:15:16,074 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:15:16,076 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce7a3c3aafdab4bea219/lava-logs.yaml 2026-03-30 01:15:16,081 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:15:16,225 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:15:16,227 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce7a3c3aafdab4bea219/lava_callback.json.gz 2026-03-30 01:15:16,248 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:16,265 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce7a3c3aafdab4bea219 HTTP/1.1" 200 None 2026-03-30 01:15:16,267 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:16,280 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:15:16,282 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:16,310 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce7a3c3aafdab4bea219 HTTP/1.1" 200 None 2026-03-30 01:15:16,311 - lava_callback - INFO - Completed processing callback for node 69c9ce7a3c3aafdab4bea219 INFO: 172.20.0.1:41474 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:15:16,751 - lava_callback - INFO - Received callback for node 69c9ce973c3aafdab4bea27b INFO: 172.20.0.1:41488 - "POST /node/69c9ce973c3aafdab4bea27b HTTP/1.0" 202 Accepted 2026-03-30 01:15:16,757 - lava_callback - INFO - Starting async job submit for node 69c9ce973c3aafdab4bea27b 2026-03-30 01:15:16,759 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:16,781 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce973c3aafdab4bea27b HTTP/1.1" 200 None 2026-03-30 01:15:16,782 - lava_callback - DEBUG - Processing node 69c9ce973c3aafdab4bea27b: baseline-nfs-arm64 2026-03-30 01:15:16,783 - lava_callback - WARNING - Failed to upload log 2026-03-30 01:15:16,785 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:15:16,911 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:15:16,914 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce973c3aafdab4bea27b/lava-logs.yaml 2026-03-30 01:15:16,915 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:15:17,028 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:15:17,029 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce973c3aafdab4bea27b/lava_callback.json.gz 2026-03-30 01:15:17,031 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:17,045 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce973c3aafdab4bea27b HTTP/1.1" 200 None 2026-03-30 01:15:17,047 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:17,061 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:15:17,062 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:17,091 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce973c3aafdab4bea27b HTTP/1.1" 200 None 2026-03-30 01:15:17,098 - lava_callback - INFO - Completed processing callback for node 69c9ce973c3aafdab4bea27b INFO: 172.20.0.1:41496 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38838 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41236 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:15:39,185 - lava_callback - INFO - Received callback for node 69c9ce843c3aafdab4bea231 INFO: 172.20.0.1:41246 - "POST /node/69c9ce843c3aafdab4bea231 HTTP/1.0" 202 Accepted 2026-03-30 01:15:39,192 - lava_callback - INFO - Starting async job submit for node 69c9ce843c3aafdab4bea231 2026-03-30 01:15:39,214 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:39,234 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce843c3aafdab4bea231 HTTP/1.1" 200 None 2026-03-30 01:15:39,235 - lava_callback - DEBUG - Processing node 69c9ce843c3aafdab4bea231: baseline-arm64 2026-03-30 01:15:39,795 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:15:39,968 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:15:39,969 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce843c3aafdab4bea231/log.txt.gz 2026-03-30 01:15:40,187 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:15:40,384 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:15:40,386 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce843c3aafdab4bea231/lava-logs.yaml 2026-03-30 01:15:40,393 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:15:40,536 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:15:40,538 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce843c3aafdab4bea231/lava_callback.json.gz 2026-03-30 01:15:40,560 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:40,574 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce843c3aafdab4bea231 HTTP/1.1" 200 None 2026-03-30 01:15:40,576 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:40,589 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:15:40,590 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:40,615 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce843c3aafdab4bea231 HTTP/1.1" 200 None 2026-03-30 01:15:40,616 - lava_callback - INFO - Completed processing callback for node 69c9ce843c3aafdab4bea231 2026-03-30 01:15:41,690 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:41,717 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-30 01:15:45,100 - lava_callback - INFO - Received callback for node 69c9ce7d3c3aafdab4bea225 INFO: 172.20.0.1:54528 - "POST /node/69c9ce7d3c3aafdab4bea225 HTTP/1.0" 202 Accepted 2026-03-30 01:15:45,107 - lava_callback - INFO - Starting async job submit for node 69c9ce7d3c3aafdab4bea225 2026-03-30 01:15:45,138 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:45,152 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce7d3c3aafdab4bea225 HTTP/1.1" 200 None 2026-03-30 01:15:45,153 - lava_callback - DEBUG - Processing node 69c9ce7d3c3aafdab4bea225: baseline-arm64 2026-03-30 01:15:45,289 - lava_callback - WARNING - Failed to upload log 2026-03-30 01:15:45,359 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:15:45,535 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:15:45,537 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce7d3c3aafdab4bea225/lava-logs.yaml 2026-03-30 01:15:45,540 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:15:45,645 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:15:45,647 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce7d3c3aafdab4bea225/lava_callback.json.gz 2026-03-30 01:15:45,685 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:45,731 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce7d3c3aafdab4bea225 HTTP/1.1" 200 None 2026-03-30 01:15:45,734 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:45,761 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:15:45,764 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:45,787 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce7d3c3aafdab4bea225 HTTP/1.1" 200 None 2026-03-30 01:15:45,867 - lava_callback - INFO - Completed processing callback for node 69c9ce7d3c3aafdab4bea225 INFO: 172.20.0.1:54532 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:15:50,351 - lava_callback - INFO - Received callback for node 69c9ce7f3c3aafdab4bea228 INFO: 172.20.0.1:60374 - "POST /node/69c9ce7f3c3aafdab4bea228 HTTP/1.0" 202 Accepted 2026-03-30 01:15:50,363 - lava_callback - INFO - Starting async job submit for node 69c9ce7f3c3aafdab4bea228 2026-03-30 01:15:50,418 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:50,440 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce7f3c3aafdab4bea228 HTTP/1.1" 200 None 2026-03-30 01:15:50,441 - lava_callback - DEBUG - Processing node 69c9ce7f3c3aafdab4bea228: baseline-arm64 2026-03-30 01:15:50,667 - lava_callback - WARNING - Failed to upload log 2026-03-30 01:15:50,784 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:15:50,957 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:15:50,959 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce7f3c3aafdab4bea228/lava-logs.yaml 2026-03-30 01:15:50,964 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:15:51,085 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:15:51,087 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce7f3c3aafdab4bea228/lava_callback.json.gz 2026-03-30 01:15:51,127 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:51,140 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce7f3c3aafdab4bea228 HTTP/1.1" 200 None 2026-03-30 01:15:51,142 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:51,156 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:15:51,157 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:51,178 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce7f3c3aafdab4bea228 HTTP/1.1" 200 None 2026-03-30 01:15:51,250 - lava_callback - INFO - Completed processing callback for node 69c9ce7f3c3aafdab4bea228 2026-03-30 01:15:51,760 - lava_callback - INFO - Received callback for node 69c9ce7b3c3aafdab4bea21f INFO: 172.20.0.1:60382 - "POST /node/69c9ce7b3c3aafdab4bea21f HTTP/1.0" 202 Accepted 2026-03-30 01:15:51,770 - lava_callback - INFO - Starting async job submit for node 69c9ce7b3c3aafdab4bea21f 2026-03-30 01:15:51,809 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:51,831 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce7b3c3aafdab4bea21f HTTP/1.1" 200 None 2026-03-30 01:15:51,833 - lava_callback - DEBUG - Processing node 69c9ce7b3c3aafdab4bea21f: baseline-arm64 2026-03-30 01:15:52,153 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:15:52,272 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:15:52,274 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce7b3c3aafdab4bea21f/log.txt.gz 2026-03-30 01:15:52,453 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:15:52,651 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:15:52,653 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce7b3c3aafdab4bea21f/lava-logs.yaml 2026-03-30 01:15:52,660 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:15:52,792 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:15:52,794 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce7b3c3aafdab4bea21f/lava_callback.json.gz 2026-03-30 01:15:52,816 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:52,831 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce7b3c3aafdab4bea21f HTTP/1.1" 200 None 2026-03-30 01:15:52,833 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:52,848 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:15:52,849 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:15:52,873 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce7b3c3aafdab4bea21f HTTP/1.1" 200 None 2026-03-30 01:15:52,874 - lava_callback - INFO - Completed processing callback for node 69c9ce7b3c3aafdab4bea21f INFO: 172.20.0.1:60384 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:16:01,970 - lava_callback - INFO - Received callback for node 69c9ce823c3aafdab4bea22e INFO: 172.20.0.1:36612 - "POST /node/69c9ce823c3aafdab4bea22e HTTP/1.0" 202 Accepted 2026-03-30 01:16:01,976 - lava_callback - INFO - Starting async job submit for node 69c9ce823c3aafdab4bea22e 2026-03-30 01:16:01,997 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:02,014 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce823c3aafdab4bea22e HTTP/1.1" 200 None 2026-03-30 01:16:02,015 - lava_callback - DEBUG - Processing node 69c9ce823c3aafdab4bea22e: baseline-arm64 2026-03-30 01:16:02,285 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:02,422 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:02,424 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce823c3aafdab4bea22e/log.txt.gz 2026-03-30 01:16:02,624 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:02,825 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:02,827 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce823c3aafdab4bea22e/lava-logs.yaml 2026-03-30 01:16:02,833 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:02,984 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:02,986 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce823c3aafdab4bea22e/lava_callback.json.gz 2026-03-30 01:16:03,010 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:03,028 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce823c3aafdab4bea22e HTTP/1.1" 200 None 2026-03-30 01:16:03,029 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:03,055 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:16:03,057 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:03,083 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce823c3aafdab4bea22e HTTP/1.1" 200 None 2026-03-30 01:16:03,088 - lava_callback - INFO - Completed processing callback for node 69c9ce823c3aafdab4bea22e 2026-03-30 01:16:04,089 - lava_callback - INFO - Received callback for node 69c9ce883c3aafdab4bea23d INFO: 172.20.0.1:36622 - "POST /node/69c9ce883c3aafdab4bea23d HTTP/1.0" 202 Accepted 2026-03-30 01:16:04,097 - lava_callback - INFO - Starting async job submit for node 69c9ce883c3aafdab4bea23d 2026-03-30 01:16:04,120 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:04,133 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce883c3aafdab4bea23d HTTP/1.1" 200 None 2026-03-30 01:16:04,134 - lava_callback - DEBUG - Processing node 69c9ce883c3aafdab4bea23d: baseline-arm64 2026-03-30 01:16:04,396 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:04,524 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:04,526 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce883c3aafdab4bea23d/log.txt.gz 2026-03-30 01:16:04,696 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:04,883 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:04,885 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce883c3aafdab4bea23d/lava-logs.yaml 2026-03-30 01:16:04,892 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:05,037 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:05,039 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce883c3aafdab4bea23d/lava_callback.json.gz 2026-03-30 01:16:05,069 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:05,083 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce883c3aafdab4bea23d HTTP/1.1" 200 None 2026-03-30 01:16:05,084 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:05,116 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:16:05,117 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:05,141 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce883c3aafdab4bea23d HTTP/1.1" 200 None 2026-03-30 01:16:05,145 - lava_callback - INFO - Completed processing callback for node 69c9ce883c3aafdab4bea23d 2026-03-30 01:16:06,358 - lava_callback - INFO - Received callback for node 69c9ce853c3aafdab4bea234 INFO: 172.20.0.1:36638 - "POST /node/69c9ce853c3aafdab4bea234 HTTP/1.0" 202 Accepted 2026-03-30 01:16:06,368 - lava_callback - INFO - Starting async job submit for node 69c9ce853c3aafdab4bea234 2026-03-30 01:16:06,388 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:06,402 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce853c3aafdab4bea234 HTTP/1.1" 200 None 2026-03-30 01:16:06,402 - lava_callback - DEBUG - Processing node 69c9ce853c3aafdab4bea234: baseline-arm64 INFO: 172.20.0.1:36640 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:16:06,774 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:06,892 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:06,894 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce853c3aafdab4bea234/log.txt.gz 2026-03-30 01:16:07,106 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:07,295 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:07,297 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce853c3aafdab4bea234/lava-logs.yaml 2026-03-30 01:16:07,305 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:07,428 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:07,430 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce853c3aafdab4bea234/lava_callback.json.gz 2026-03-30 01:16:07,450 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:07,465 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce853c3aafdab4bea234 HTTP/1.1" 200 None 2026-03-30 01:16:07,466 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:07,480 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:16:07,482 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:07,515 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce853c3aafdab4bea234 HTTP/1.1" 200 None 2026-03-30 01:16:07,517 - lava_callback - INFO - Completed processing callback for node 69c9ce853c3aafdab4bea234 INFO: 172.20.0.1:36646 - "GET /node/6983aa6766dc8a56088a705a%5Cn HTTP/1.0" 405 Method Not Allowed 2026-03-30 01:16:11,719 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:11,736 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 2026-03-30 01:16:13,585 - lava_callback - INFO - Received callback for node 69c9ce8c3c3aafdab4bea246 INFO: 172.20.0.1:41186 - "POST /node/69c9ce8c3c3aafdab4bea246 HTTP/1.0" 202 Accepted 2026-03-30 01:16:13,596 - lava_callback - INFO - Starting async job submit for node 69c9ce8c3c3aafdab4bea246 2026-03-30 01:16:13,635 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:13,662 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce8c3c3aafdab4bea246 HTTP/1.1" 200 None 2026-03-30 01:16:13,664 - lava_callback - DEBUG - Processing node 69c9ce8c3c3aafdab4bea246: baseline-arm64 2026-03-30 01:16:14,093 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:14,224 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:14,226 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce8c3c3aafdab4bea246/log.txt.gz 2026-03-30 01:16:14,412 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:14,633 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:14,635 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce8c3c3aafdab4bea246/lava-logs.yaml 2026-03-30 01:16:14,643 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:14,778 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:14,779 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce8c3c3aafdab4bea246/lava_callback.json.gz 2026-03-30 01:16:14,802 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:14,817 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce8c3c3aafdab4bea246 HTTP/1.1" 200 None 2026-03-30 01:16:14,819 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:14,833 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:16:14,834 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:14,870 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce8c3c3aafdab4bea246 HTTP/1.1" 200 None 2026-03-30 01:16:14,873 - lava_callback - INFO - Completed processing callback for node 69c9ce8c3c3aafdab4bea246 2026-03-30 01:16:14,886 - lava_callback - INFO - Received callback for node 69c9ce8b3c3aafdab4bea243 INFO: 172.20.0.1:41202 - "POST /node/69c9ce8b3c3aafdab4bea243 HTTP/1.0" 202 Accepted 2026-03-30 01:16:14,921 - lava_callback - INFO - Starting async job submit for node 69c9ce8b3c3aafdab4bea243 2026-03-30 01:16:14,969 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:14,987 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce8b3c3aafdab4bea243 HTTP/1.1" 200 None 2026-03-30 01:16:14,988 - lava_callback - DEBUG - Processing node 69c9ce8b3c3aafdab4bea243: baseline-arm64 INFO: 172.20.0.1:41206 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-30 01:16:15,311 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:15,423 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:15,425 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce8b3c3aafdab4bea243/log.txt.gz 2026-03-30 01:16:15,596 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:15,777 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:15,779 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce8b3c3aafdab4bea243/lava-logs.yaml 2026-03-30 01:16:15,785 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:15,915 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:15,917 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce8b3c3aafdab4bea243/lava_callback.json.gz 2026-03-30 01:16:15,940 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:15,962 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce8b3c3aafdab4bea243 HTTP/1.1" 200 None 2026-03-30 01:16:15,964 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:15,983 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:16:15,985 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:16,012 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce8b3c3aafdab4bea243 HTTP/1.1" 200 None 2026-03-30 01:16:16,013 - lava_callback - INFO - Completed processing callback for node 69c9ce8b3c3aafdab4bea243 INFO: 172.20.0.1:41220 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41236 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:16:22,018 - lava_callback - INFO - Received callback for node 69c9ce7a3c3aafdab4bea21c INFO: 172.20.0.1:39084 - "POST /node/69c9ce7a3c3aafdab4bea21c HTTP/1.0" 202 Accepted 2026-03-30 01:16:22,024 - lava_callback - INFO - Starting async job submit for node 69c9ce7a3c3aafdab4bea21c 2026-03-30 01:16:22,045 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:22,060 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce7a3c3aafdab4bea21c HTTP/1.1" 200 None 2026-03-30 01:16:22,060 - lava_callback - DEBUG - Processing node 69c9ce7a3c3aafdab4bea21c: baseline-arm64 2026-03-30 01:16:22,315 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:22,433 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:22,435 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce7a3c3aafdab4bea21c/log.txt.gz 2026-03-30 01:16:22,567 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:22,738 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:22,740 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce7a3c3aafdab4bea21c/lava-logs.yaml 2026-03-30 01:16:22,746 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:22,882 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:22,883 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce7a3c3aafdab4bea21c/lava_callback.json.gz 2026-03-30 01:16:22,906 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:22,938 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce7a3c3aafdab4bea21c HTTP/1.1" 200 None 2026-03-30 01:16:22,939 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:22,952 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:16:22,954 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:22,978 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce7a3c3aafdab4bea21c HTTP/1.1" 200 None 2026-03-30 01:16:22,987 - lava_callback - INFO - Completed processing callback for node 69c9ce7a3c3aafdab4bea21c INFO: 172.20.0.1:39098 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:16:27,056 - lava_callback - INFO - Received callback for node 69c9ce7c3c3aafdab4bea222 INFO: 172.20.0.1:39104 - "POST /node/69c9ce7c3c3aafdab4bea222 HTTP/1.0" 202 Accepted 2026-03-30 01:16:27,062 - lava_callback - INFO - Starting async job submit for node 69c9ce7c3c3aafdab4bea222 2026-03-30 01:16:27,084 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:27,098 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce7c3c3aafdab4bea222 HTTP/1.1" 200 None 2026-03-30 01:16:27,098 - lava_callback - DEBUG - Processing node 69c9ce7c3c3aafdab4bea222: baseline-arm64 2026-03-30 01:16:27,344 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:27,452 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:27,453 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce7c3c3aafdab4bea222/log.txt.gz 2026-03-30 01:16:27,610 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:27,800 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:27,802 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce7c3c3aafdab4bea222/lava-logs.yaml 2026-03-30 01:16:27,808 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:27,936 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:27,938 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce7c3c3aafdab4bea222/lava_callback.json.gz 2026-03-30 01:16:27,961 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:27,975 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce7c3c3aafdab4bea222 HTTP/1.1" 200 None 2026-03-30 01:16:27,976 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:27,991 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:16:27,992 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:28,021 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce7c3c3aafdab4bea222 HTTP/1.1" 200 None 2026-03-30 01:16:28,023 - lava_callback - INFO - Completed processing callback for node 69c9ce7c3c3aafdab4bea222 2026-03-30 01:16:31,509 - lava_callback - INFO - Received callback for node 69c9ce8a3c3aafdab4bea240 INFO: 172.20.0.1:59880 - "POST /node/69c9ce8a3c3aafdab4bea240 HTTP/1.0" 202 Accepted 2026-03-30 01:16:31,515 - lava_callback - INFO - Starting async job submit for node 69c9ce8a3c3aafdab4bea240 2026-03-30 01:16:31,537 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:31,565 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce8a3c3aafdab4bea240 HTTP/1.1" 200 None 2026-03-30 01:16:31,566 - lava_callback - DEBUG - Processing node 69c9ce8a3c3aafdab4bea240: baseline-arm64 2026-03-30 01:16:31,836 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:31,948 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:31,950 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce8a3c3aafdab4bea240/log.txt.gz 2026-03-30 01:16:32,152 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:32,357 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:32,359 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce8a3c3aafdab4bea240/lava-logs.yaml 2026-03-30 01:16:32,366 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:32,500 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:32,502 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce8a3c3aafdab4bea240/lava_callback.json.gz 2026-03-30 01:16:32,532 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:32,550 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce8a3c3aafdab4bea240 HTTP/1.1" 200 None 2026-03-30 01:16:32,552 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:32,565 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:16:32,567 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:32,590 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce8a3c3aafdab4bea240 HTTP/1.1" 200 None 2026-03-30 01:16:32,594 - lava_callback - INFO - Completed processing callback for node 69c9ce8a3c3aafdab4bea240 2026-03-30 01:16:33,077 - lava_callback - INFO - Received callback for node 69c9ce943c3aafdab4bea275 INFO: 172.20.0.1:59892 - "POST /node/69c9ce943c3aafdab4bea275 HTTP/1.0" 202 Accepted 2026-03-30 01:16:33,084 - lava_callback - INFO - Starting async job submit for node 69c9ce943c3aafdab4bea275 2026-03-30 01:16:33,114 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:33,130 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce943c3aafdab4bea275 HTTP/1.1" 200 None 2026-03-30 01:16:33,131 - lava_callback - DEBUG - Processing node 69c9ce943c3aafdab4bea275: baseline-nfs-arm64 2026-03-30 01:16:33,589 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:33,732 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:33,734 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce943c3aafdab4bea275/log.txt.gz 2026-03-30 01:16:34,016 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:34,223 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:34,225 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce943c3aafdab4bea275/lava-logs.yaml 2026-03-30 01:16:34,235 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:34,391 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:34,394 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce943c3aafdab4bea275/lava_callback.json.gz 2026-03-30 01:16:34,422 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:34,436 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce943c3aafdab4bea275 HTTP/1.1" 200 None 2026-03-30 01:16:34,437 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:34,455 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:16:34,457 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:34,493 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce943c3aafdab4bea275 HTTP/1.1" 200 None 2026-03-30 01:16:34,494 - lava_callback - INFO - Completed processing callback for node 69c9ce943c3aafdab4bea275 2026-03-30 01:16:35,988 - lava_callback - INFO - Received callback for node 69c9ce9b3c3aafdab4bea284 INFO: 172.20.0.1:59898 - "POST /node/69c9ce9b3c3aafdab4bea284 HTTP/1.0" 202 Accepted 2026-03-30 01:16:35,994 - lava_callback - INFO - Starting async job submit for node 69c9ce9b3c3aafdab4bea284 2026-03-30 01:16:36,018 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:36,032 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce9b3c3aafdab4bea284 HTTP/1.1" 200 None 2026-03-30 01:16:36,033 - lava_callback - DEBUG - Processing node 69c9ce9b3c3aafdab4bea284: baseline-nfs-arm64 2026-03-30 01:16:36,401 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.20.0.1:59904 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:16:36,556 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:36,558 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce9b3c3aafdab4bea284/log.txt.gz 2026-03-30 01:16:36,582 - lava_callback - INFO - Received callback for node 69c9ce953c3aafdab4bea278 INFO: 172.20.0.1:59916 - "POST /node/69c9ce953c3aafdab4bea278 HTTP/1.0" 202 Accepted 2026-03-30 01:16:36,609 - lava_callback - INFO - Starting async job submit for node 69c9ce953c3aafdab4bea278 2026-03-30 01:16:36,663 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:36,725 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce953c3aafdab4bea278 HTTP/1.1" 200 None 2026-03-30 01:16:36,736 - lava_callback - DEBUG - Processing node 69c9ce953c3aafdab4bea278: baseline-nfs-arm64 2026-03-30 01:16:36,925 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:37,251 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:37,298 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce9b3c3aafdab4bea284/lava-logs.yaml 2026-03-30 01:16:37,374 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:37,378 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:37,532 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:37,535 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce9b3c3aafdab4bea284/lava_callback.json.gz 2026-03-30 01:16:37,571 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:37,572 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:37,573 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce953c3aafdab4bea278/log.txt.gz 2026-03-30 01:16:37,628 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce9b3c3aafdab4bea284 HTTP/1.1" 200 None 2026-03-30 01:16:37,653 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:37,731 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:16:37,754 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:37,847 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce9b3c3aafdab4bea284 HTTP/1.1" 200 None 2026-03-30 01:16:37,854 - lava_callback - INFO - Completed processing callback for node 69c9ce9b3c3aafdab4bea284 2026-03-30 01:16:37,973 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:38,222 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:38,224 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce953c3aafdab4bea278/lava-logs.yaml 2026-03-30 01:16:38,236 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:38,383 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:38,384 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce953c3aafdab4bea278/lava_callback.json.gz 2026-03-30 01:16:38,409 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:38,423 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce953c3aafdab4bea278 HTTP/1.1" 200 None 2026-03-30 01:16:38,425 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:38,439 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:16:38,441 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:38,465 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce953c3aafdab4bea278 HTTP/1.1" 200 None 2026-03-30 01:16:38,466 - lava_callback - INFO - Completed processing callback for node 69c9ce953c3aafdab4bea278 2026-03-30 01:16:41,738 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:41,763 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 2026-03-30 01:16:42,109 - lava_callback - INFO - Received callback for node 69c9cec33c3aafdab4bea312 INFO: 172.20.0.1:54982 - "POST /node/69c9cec33c3aafdab4bea312 HTTP/1.0" 202 Accepted 2026-03-30 01:16:42,118 - lava_callback - INFO - Starting async job submit for node 69c9cec33c3aafdab4bea312 2026-03-30 01:16:42,143 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:42,158 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cec33c3aafdab4bea312 HTTP/1.1" 200 None 2026-03-30 01:16:42,159 - lava_callback - DEBUG - Processing node 69c9cec33c3aafdab4bea312: baseline-arm64-foundriesio 2026-03-30 01:16:42,395 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:42,516 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:42,518 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69c9cec33c3aafdab4bea312/log.txt.gz 2026-03-30 01:16:42,660 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:42,884 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:42,886 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69c9cec33c3aafdab4bea312/lava-logs.yaml 2026-03-30 01:16:42,893 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:43,039 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:43,041 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69c9cec33c3aafdab4bea312/lava_callback.json.gz 2026-03-30 01:16:43,065 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:43,088 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cec33c3aafdab4bea312 HTTP/1.1" 200 None 2026-03-30 01:16:43,090 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:43,106 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:16:43,108 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:43,134 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9cec33c3aafdab4bea312 HTTP/1.1" 200 None 2026-03-30 01:16:43,135 - lava_callback - INFO - Completed processing callback for node 69c9cec33c3aafdab4bea312 2026-03-30 01:16:44,253 - lava_callback - INFO - Received callback for node 69c9ce9a3c3aafdab4bea281 INFO: 172.20.0.1:54998 - "POST /node/69c9ce9a3c3aafdab4bea281 HTTP/1.0" 202 Accepted 2026-03-30 01:16:44,264 - lava_callback - INFO - Starting async job submit for node 69c9ce9a3c3aafdab4bea281 2026-03-30 01:16:44,290 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:44,309 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce9a3c3aafdab4bea281 HTTP/1.1" 200 None 2026-03-30 01:16:44,310 - lava_callback - DEBUG - Processing node 69c9ce9a3c3aafdab4bea281: baseline-nfs-arm64 2026-03-30 01:16:44,696 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:44,842 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:44,844 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce9a3c3aafdab4bea281/log.txt.gz 2026-03-30 01:16:45,067 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:45,258 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:45,260 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce9a3c3aafdab4bea281/lava-logs.yaml 2026-03-30 01:16:45,268 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:45,418 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:45,420 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce9a3c3aafdab4bea281/lava_callback.json.gz 2026-03-30 01:16:45,443 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:45,457 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce9a3c3aafdab4bea281 HTTP/1.1" 200 None 2026-03-30 01:16:45,459 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:45,472 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:16:45,473 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:45,500 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce9a3c3aafdab4bea281 HTTP/1.1" 200 None 2026-03-30 01:16:45,505 - lava_callback - INFO - Completed processing callback for node 69c9ce9a3c3aafdab4bea281 2026-03-30 01:16:45,912 - lava_callback - INFO - Received callback for node 69c72be929015d0f9599225a INFO: 172.20.0.1:55000 - "POST /node/69c72be929015d0f9599225a HTTP/1.0" 202 Accepted 2026-03-30 01:16:45,918 - lava_callback - INFO - Starting async job submit for node 69c72be929015d0f9599225a 2026-03-30 01:16:45,919 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:45,934 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72be929015d0f9599225a HTTP/1.1" 200 None 2026-03-30 01:16:45,935 - lava_callback - DEBUG - Processing node 69c72be929015d0f9599225a: baseline-arm-clabbe 2026-03-30 01:16:45,935 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72be929015d0f9599225a 2026-03-30 01:16:45,935 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:45,950 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72be929015d0f9599225a HTTP/1.1" 200 None 2026-03-30 01:16:45,952 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:45,966 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69bf353af1004e82292e0db7 HTTP/1.1" 200 None 2026-03-30 01:16:45,967 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:45,987 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72be929015d0f9599225a HTTP/1.1" 200 None 2026-03-30 01:16:45,988 - lava_callback - ERROR - Error processing callback for node 69c72be929015d0f9599225a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55002 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:16:48,693 - lava_callback - INFO - Received callback for node 69c9ce9c3c3aafdab4bea287 INFO: 172.20.0.1:55006 - "POST /node/69c9ce9c3c3aafdab4bea287 HTTP/1.0" 202 Accepted 2026-03-30 01:16:48,699 - lava_callback - INFO - Starting async job submit for node 69c9ce9c3c3aafdab4bea287 2026-03-30 01:16:48,722 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:48,737 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce9c3c3aafdab4bea287 HTTP/1.1" 200 None 2026-03-30 01:16:48,738 - lava_callback - DEBUG - Processing node 69c9ce9c3c3aafdab4bea287: baseline-nfs-arm64 INFO: 172.20.0.1:55008 - "GET /node/69830f39409235c29c0de6f1%5Cn HTTP/1.0" 405 Method Not Allowed 2026-03-30 01:16:49,141 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:49,271 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:49,273 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce9c3c3aafdab4bea287/log.txt.gz 2026-03-30 01:16:49,505 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:49,712 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:49,715 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce9c3c3aafdab4bea287/lava-logs.yaml 2026-03-30 01:16:49,724 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:49,875 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:49,877 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce9c3c3aafdab4bea287/lava_callback.json.gz 2026-03-30 01:16:49,901 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:49,917 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce9c3c3aafdab4bea287 HTTP/1.1" 200 None 2026-03-30 01:16:49,918 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:49,932 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:16:49,933 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:49,966 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce9c3c3aafdab4bea287 HTTP/1.1" 200 None 2026-03-30 01:16:49,967 - lava_callback - INFO - Completed processing callback for node 69c9ce9c3c3aafdab4bea287 2026-03-30 01:16:55,737 - lava_callback - INFO - Received callback for node 69c9ce913c3aafdab4bea26f INFO: 172.20.0.1:34478 - "POST /node/69c9ce913c3aafdab4bea26f HTTP/1.0" 202 Accepted 2026-03-30 01:16:55,744 - lava_callback - INFO - Starting async job submit for node 69c9ce913c3aafdab4bea26f 2026-03-30 01:16:55,766 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:55,790 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce913c3aafdab4bea26f HTTP/1.1" 200 None 2026-03-30 01:16:55,791 - lava_callback - DEBUG - Processing node 69c9ce913c3aafdab4bea26f: baseline-nfs-arm64 2026-03-30 01:16:56,149 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:56,281 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:56,283 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce913c3aafdab4bea26f/log.txt.gz 2026-03-30 01:16:56,478 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.20.0.1:34492 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:16:56,698 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:56,700 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce913c3aafdab4bea26f/lava-logs.yaml 2026-03-30 01:16:56,708 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:56,845 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:56,847 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce913c3aafdab4bea26f/lava_callback.json.gz 2026-03-30 01:16:56,872 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:56,890 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce913c3aafdab4bea26f HTTP/1.1" 200 None 2026-03-30 01:16:56,891 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:56,908 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:16:56,909 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:56,935 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce913c3aafdab4bea26f HTTP/1.1" 200 None 2026-03-30 01:16:56,936 - lava_callback - INFO - Completed processing callback for node 69c9ce913c3aafdab4bea26f 2026-03-30 01:16:59,228 - lava_callback - INFO - Received callback for node 69c9cec03c3aafdab4bea30a INFO: 172.20.0.1:34500 - "POST /node/69c9cec03c3aafdab4bea30a HTTP/1.0" 202 Accepted 2026-03-30 01:16:59,235 - lava_callback - INFO - Starting async job submit for node 69c9cec03c3aafdab4bea30a 2026-03-30 01:16:59,255 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:59,256 - lava_callback - INFO - Received callback for node 69c9ceb13c3aafdab4bea2db INFO: 172.20.0.1:34504 - "POST /node/69c9ceb13c3aafdab4bea2db HTTP/1.0" 202 Accepted 2026-03-30 01:16:59,262 - lava_callback - INFO - Starting async job submit for node 69c9ceb13c3aafdab4bea2db 2026-03-30 01:16:59,285 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:16:59,297 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cec03c3aafdab4bea30a HTTP/1.1" 200 None 2026-03-30 01:16:59,298 - lava_callback - DEBUG - Processing node 69c9cec03c3aafdab4bea30a: baseline-arm64-pengutronix 2026-03-30 01:16:59,309 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ceb13c3aafdab4bea2db HTTP/1.1" 200 None 2026-03-30 01:16:59,325 - lava_callback - DEBUG - Processing node 69c9ceb13c3aafdab4bea2db: baseline-arm64-broonie 2026-03-30 01:16:59,744 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:59,747 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:16:59,886 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:59,887 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c9cec03c3aafdab4bea30a/log.txt.gz 2026-03-30 01:16:59,889 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:16:59,934 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceb13c3aafdab4bea2db/log.txt.gz 2026-03-30 01:17:00,131 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:00,195 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:00,386 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:00,388 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c9cec03c3aafdab4bea30a/lava-logs.yaml 2026-03-30 01:17:00,395 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:00,407 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:00,409 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceb13c3aafdab4bea2db/lava-logs.yaml 2026-03-30 01:17:00,415 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:00,550 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:00,551 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c9cec03c3aafdab4bea30a/lava_callback.json.gz 2026-03-30 01:17:00,576 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:00,577 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:00,578 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceb13c3aafdab4bea2db/lava_callback.json.gz 2026-03-30 01:17:00,602 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:00,613 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cec03c3aafdab4bea30a HTTP/1.1" 200 None 2026-03-30 01:17:00,615 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:00,618 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ceb13c3aafdab4bea2db HTTP/1.1" 200 None 2026-03-30 01:17:00,620 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:00,635 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:17:00,636 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:00,638 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:17:00,640 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:00,669 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9cec03c3aafdab4bea30a HTTP/1.1" 200 None 2026-03-30 01:17:00,677 - lava_callback - INFO - Completed processing callback for node 69c9cec03c3aafdab4bea30a 2026-03-30 01:17:00,684 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ceb13c3aafdab4bea2db HTTP/1.1" 200 None 2026-03-30 01:17:00,686 - lava_callback - INFO - Completed processing callback for node 69c9ceb13c3aafdab4bea2db 2026-03-30 01:17:02,735 - lava_callback - INFO - Received callback for node 69c9ce983c3aafdab4bea27e INFO: 172.20.0.1:49594 - "POST /node/69c9ce983c3aafdab4bea27e HTTP/1.0" 202 Accepted 2026-03-30 01:17:02,742 - lava_callback - INFO - Starting async job submit for node 69c9ce983c3aafdab4bea27e 2026-03-30 01:17:02,766 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:02,789 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce983c3aafdab4bea27e HTTP/1.1" 200 None 2026-03-30 01:17:02,790 - lava_callback - DEBUG - Processing node 69c9ce983c3aafdab4bea27e: baseline-nfs-arm64 2026-03-30 01:17:03,115 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:03,235 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:03,237 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce983c3aafdab4bea27e/log.txt.gz 2026-03-30 01:17:03,429 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:03,623 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:03,625 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce983c3aafdab4bea27e/lava-logs.yaml 2026-03-30 01:17:03,633 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:03,767 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:03,768 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce983c3aafdab4bea27e/lava_callback.json.gz 2026-03-30 01:17:03,792 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:03,810 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce983c3aafdab4bea27e HTTP/1.1" 200 None 2026-03-30 01:17:03,812 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:03,825 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:17:03,827 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:03,851 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce983c3aafdab4bea27e HTTP/1.1" 200 None 2026-03-30 01:17:03,853 - lava_callback - INFO - Completed processing callback for node 69c9ce983c3aafdab4bea27e 2026-03-30 01:17:06,420 - lava_callback - INFO - Received callback for node 69c9ceb43c3aafdab4bea2e1 INFO: 172.20.0.1:49596 - "POST /node/69c9ceb43c3aafdab4bea2e1 HTTP/1.0" 202 Accepted 2026-03-30 01:17:06,426 - lava_callback - INFO - Starting async job submit for node 69c9ceb43c3aafdab4bea2e1 2026-03-30 01:17:06,449 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:06,464 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ceb43c3aafdab4bea2e1 HTTP/1.1" 200 None 2026-03-30 01:17:06,465 - lava_callback - DEBUG - Processing node 69c9ceb43c3aafdab4bea2e1: baseline-arm64-broonie INFO: 172.20.0.1:49598 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:17:06,707 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:06,822 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:06,824 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceb43c3aafdab4bea2e1/log.txt.gz 2026-03-30 01:17:06,993 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:07,175 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:07,177 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceb43c3aafdab4bea2e1/lava-logs.yaml 2026-03-30 01:17:07,183 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:07,320 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:07,322 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceb43c3aafdab4bea2e1/lava_callback.json.gz 2026-03-30 01:17:07,345 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:07,364 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ceb43c3aafdab4bea2e1 HTTP/1.1" 200 None 2026-03-30 01:17:07,365 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:07,379 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:17:07,381 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:07,411 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ceb43c3aafdab4bea2e1 HTTP/1.1" 200 None 2026-03-30 01:17:07,412 - lava_callback - INFO - Completed processing callback for node 69c9ceb43c3aafdab4bea2e1 2026-03-30 01:17:08,870 - lava_callback - INFO - Received callback for node 69c9ce8f3c3aafdab4bea24c INFO: 172.20.0.1:49600 - "POST /node/69c9ce8f3c3aafdab4bea24c HTTP/1.0" 202 Accepted 2026-03-30 01:17:08,877 - lava_callback - INFO - Starting async job submit for node 69c9ce8f3c3aafdab4bea24c 2026-03-30 01:17:08,899 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:08,913 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce8f3c3aafdab4bea24c HTTP/1.1" 200 None 2026-03-30 01:17:08,914 - lava_callback - DEBUG - Processing node 69c9ce8f3c3aafdab4bea24c: baseline-nfs-arm64 2026-03-30 01:17:09,232 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:09,367 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:09,369 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce8f3c3aafdab4bea24c/log.txt.gz 2026-03-30 01:17:09,553 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:09,758 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:09,760 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce8f3c3aafdab4bea24c/lava-logs.yaml 2026-03-30 01:17:09,768 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:09,901 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:09,902 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce8f3c3aafdab4bea24c/lava_callback.json.gz 2026-03-30 01:17:09,926 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:09,940 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce8f3c3aafdab4bea24c HTTP/1.1" 200 None 2026-03-30 01:17:09,942 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:09,957 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:17:09,958 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:09,983 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce8f3c3aafdab4bea24c HTTP/1.1" 200 None 2026-03-30 01:17:09,989 - lava_callback - INFO - Completed processing callback for node 69c9ce8f3c3aafdab4bea24c 2026-03-30 01:17:10,065 - lava_callback - INFO - Received callback for node 69c9cec43c3aafdab4bea315 INFO: 172.20.0.1:42454 - "POST /node/69c9cec43c3aafdab4bea315 HTTP/1.0" 202 Accepted 2026-03-30 01:17:10,076 - lava_callback - INFO - Starting async job submit for node 69c9cec43c3aafdab4bea315 2026-03-30 01:17:10,120 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:10,144 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cec43c3aafdab4bea315 HTTP/1.1" 200 None 2026-03-30 01:17:10,145 - lava_callback - DEBUG - Processing node 69c9cec43c3aafdab4bea315: baseline-arm64-foundriesio 2026-03-30 01:17:10,643 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:10,775 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:10,776 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69c9cec43c3aafdab4bea315/log.txt.gz 2026-03-30 01:17:11,028 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:11,221 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:11,224 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69c9cec43c3aafdab4bea315/lava-logs.yaml 2026-03-30 01:17:11,234 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:11,384 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:11,385 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69c9cec43c3aafdab4bea315/lava_callback.json.gz 2026-03-30 01:17:11,423 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:11,437 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cec43c3aafdab4bea315 HTTP/1.1" 200 None 2026-03-30 01:17:11,439 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:11,456 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:17:11,457 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:11,481 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9cec43c3aafdab4bea315 HTTP/1.1" 200 None 2026-03-30 01:17:11,482 - lava_callback - INFO - Completed processing callback for node 69c9cec43c3aafdab4bea315 2026-03-30 01:17:11,765 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:11,786 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 2026-03-30 01:17:14,204 - lava_callback - INFO - Received callback for node 69c9ce863c3aafdab4bea237 INFO: 172.20.0.1:42470 - "POST /node/69c9ce863c3aafdab4bea237 HTTP/1.0" 202 Accepted 2026-03-30 01:17:14,211 - lava_callback - INFO - Starting async job submit for node 69c9ce863c3aafdab4bea237 2026-03-30 01:17:14,233 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:14,248 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce863c3aafdab4bea237 HTTP/1.1" 200 None 2026-03-30 01:17:14,249 - lava_callback - DEBUG - Processing node 69c9ce863c3aafdab4bea237: baseline-arm64 2026-03-30 01:17:14,692 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:14,819 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:14,821 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce863c3aafdab4bea237/log.txt.gz 2026-03-30 01:17:15,097 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.20.0.1:42482 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-30 01:17:15,330 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:15,332 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce863c3aafdab4bea237/lava-logs.yaml 2026-03-30 01:17:15,342 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:15,548 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:15,550 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce863c3aafdab4bea237/lava_callback.json.gz 2026-03-30 01:17:15,595 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:15,609 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce863c3aafdab4bea237 HTTP/1.1" 200 None 2026-03-30 01:17:15,611 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:15,625 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:17:15,626 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:15,657 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce863c3aafdab4bea237 HTTP/1.1" 200 None 2026-03-30 01:17:15,715 - lava_callback - INFO - Completed processing callback for node 69c9ce863c3aafdab4bea237 INFO: 172.20.0.1:42492 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42502 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:17:23,483 - lava_callback - INFO - Received callback for node 69c9cebb3c3aafdab4bea2f3 INFO: 172.20.0.1:59008 - "POST /node/69c9cebb3c3aafdab4bea2f3 HTTP/1.0" 202 Accepted 2026-03-30 01:17:23,489 - lava_callback - INFO - Starting async job submit for node 69c9cebb3c3aafdab4bea2f3 2026-03-30 01:17:23,510 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:23,526 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cebb3c3aafdab4bea2f3 HTTP/1.1" 200 None 2026-03-30 01:17:23,526 - lava_callback - DEBUG - Processing node 69c9cebb3c3aafdab4bea2f3: baseline-arm64-clabbe 2026-03-30 01:17:23,755 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:23,865 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:23,867 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69c9cebb3c3aafdab4bea2f3/log.txt.gz 2026-03-30 01:17:24,003 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:24,178 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:24,180 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69c9cebb3c3aafdab4bea2f3/lava-logs.yaml 2026-03-30 01:17:24,186 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:24,308 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:24,310 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69c9cebb3c3aafdab4bea2f3/lava_callback.json.gz 2026-03-30 01:17:24,332 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:24,347 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cebb3c3aafdab4bea2f3 HTTP/1.1" 200 None 2026-03-30 01:17:24,349 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:24,363 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:17:24,365 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:24,397 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9cebb3c3aafdab4bea2f3 HTTP/1.1" 200 None 2026-03-30 01:17:24,402 - lava_callback - INFO - Completed processing callback for node 69c9cebb3c3aafdab4bea2f3 2026-03-30 01:17:26,065 - lava_callback - INFO - Received callback for node 69c72c1129015d0f9599226f INFO: 172.20.0.1:59016 - "POST /node/69c72c1129015d0f9599226f HTTP/1.0" 202 Accepted 2026-03-30 01:17:26,070 - lava_callback - INFO - Starting async job submit for node 69c72c1129015d0f9599226f 2026-03-30 01:17:26,071 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:26,085 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72c1129015d0f9599226f HTTP/1.1" 200 None 2026-03-30 01:17:26,086 - lava_callback - DEBUG - Processing node 69c72c1129015d0f9599226f: baseline-arm-clabbe 2026-03-30 01:17:26,086 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72c1129015d0f9599226f 2026-03-30 01:17:26,087 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:26,100 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72c1129015d0f9599226f HTTP/1.1" 200 None 2026-03-30 01:17:26,102 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:26,115 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69bf353af1004e82292e0db7 HTTP/1.1" 200 None 2026-03-30 01:17:26,116 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:26,142 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72c1129015d0f9599226f HTTP/1.1" 200 None 2026-03-30 01:17:26,143 - lava_callback - ERROR - Error processing callback for node 69c72c1129015d0f9599226f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-30 01:17:26,485 - lava_callback - INFO - Received callback for node 69c9ceac3c3aafdab4bea2b6 INFO: 172.20.0.1:59032 - "POST /node/69c9ceac3c3aafdab4bea2b6 HTTP/1.0" 202 Accepted 2026-03-30 01:17:26,492 - lava_callback - INFO - Starting async job submit for node 69c9ceac3c3aafdab4bea2b6 INFO: 172.20.0.1:59044 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:17:26,516 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:26,537 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ceac3c3aafdab4bea2b6 HTTP/1.1" 200 None 2026-03-30 01:17:26,538 - lava_callback - DEBUG - Processing node 69c9ceac3c3aafdab4bea2b6: baseline-arm64-broonie 2026-03-30 01:17:26,813 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:26,924 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:26,925 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceac3c3aafdab4bea2b6/log.txt.gz 2026-03-30 01:17:27,090 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:27,290 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:27,292 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceac3c3aafdab4bea2b6/lava-logs.yaml 2026-03-30 01:17:27,299 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:27,432 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:27,434 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceac3c3aafdab4bea2b6/lava_callback.json.gz 2026-03-30 01:17:27,457 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:27,472 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ceac3c3aafdab4bea2b6 HTTP/1.1" 200 None 2026-03-30 01:17:27,474 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:27,488 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:17:27,489 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:27,515 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ceac3c3aafdab4bea2b6 HTTP/1.1" 200 None 2026-03-30 01:17:27,521 - lava_callback - INFO - Completed processing callback for node 69c9ceac3c3aafdab4bea2b6 2026-03-30 01:17:27,936 - lava_callback - INFO - Received callback for node 69c9ceae3c3aafdab4bea2b9 INFO: 172.20.0.1:59054 - "POST /node/69c9ceae3c3aafdab4bea2b9 HTTP/1.0" 202 Accepted 2026-03-30 01:17:27,942 - lava_callback - INFO - Starting async job submit for node 69c9ceae3c3aafdab4bea2b9 2026-03-30 01:17:27,966 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:27,995 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ceae3c3aafdab4bea2b9 HTTP/1.1" 200 None 2026-03-30 01:17:27,996 - lava_callback - DEBUG - Processing node 69c9ceae3c3aafdab4bea2b9: baseline-arm64-broonie 2026-03-30 01:17:28,298 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:28,409 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:28,411 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceae3c3aafdab4bea2b9/log.txt.gz 2026-03-30 01:17:28,578 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:28,793 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:28,795 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceae3c3aafdab4bea2b9/lava-logs.yaml 2026-03-30 01:17:28,802 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:28,945 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:28,947 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceae3c3aafdab4bea2b9/lava_callback.json.gz 2026-03-30 01:17:28,971 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:28,987 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ceae3c3aafdab4bea2b9 HTTP/1.1" 200 None 2026-03-30 01:17:28,989 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:29,003 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:17:29,005 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:29,032 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ceae3c3aafdab4bea2b9 HTTP/1.1" 200 None 2026-03-30 01:17:29,037 - lava_callback - INFO - Completed processing callback for node 69c9ceae3c3aafdab4bea2b9 INFO: 172.20.0.1:55176 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:17:38,978 - lava_callback - INFO - Received callback for node 69c9ceb63c3aafdab4bea2e4 INFO: 172.20.0.1:55192 - "POST /node/69c9ceb63c3aafdab4bea2e4 HTTP/1.0" 202 Accepted 2026-03-30 01:17:38,984 - lava_callback - INFO - Starting async job submit for node 69c9ceb63c3aafdab4bea2e4 2026-03-30 01:17:39,007 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:39,030 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ceb63c3aafdab4bea2e4 HTTP/1.1" 200 None 2026-03-30 01:17:39,031 - lava_callback - DEBUG - Processing node 69c9ceb63c3aafdab4bea2e4: baseline-arm64-broonie 2026-03-30 01:17:39,284 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:39,419 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:39,420 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceb63c3aafdab4bea2e4/log.txt.gz 2026-03-30 01:17:39,570 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:39,918 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:39,920 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceb63c3aafdab4bea2e4/lava-logs.yaml 2026-03-30 01:17:39,926 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:40,052 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:40,054 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceb63c3aafdab4bea2e4/lava_callback.json.gz 2026-03-30 01:17:40,078 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:40,093 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ceb63c3aafdab4bea2e4 HTTP/1.1" 200 None 2026-03-30 01:17:40,094 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:40,111 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:17:40,112 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:40,138 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ceb63c3aafdab4bea2e4 HTTP/1.1" 200 None 2026-03-30 01:17:40,144 - lava_callback - INFO - Completed processing callback for node 69c9ceb63c3aafdab4bea2e4 2026-03-30 01:17:41,788 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:41,804 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 INFO: 172.20.0.1:40604 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:17:46,530 - lava_callback - INFO - Received callback for node 69c9ceb33c3aafdab4bea2de INFO: 172.20.0.1:40606 - "POST /node/69c9ceb33c3aafdab4bea2de HTTP/1.0" 202 Accepted 2026-03-30 01:17:46,536 - lava_callback - INFO - Starting async job submit for node 69c9ceb33c3aafdab4bea2de 2026-03-30 01:17:46,558 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:46,601 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ceb33c3aafdab4bea2de HTTP/1.1" 200 None 2026-03-30 01:17:46,602 - lava_callback - DEBUG - Processing node 69c9ceb33c3aafdab4bea2de: baseline-arm64-broonie 2026-03-30 01:17:46,893 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:47,011 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:47,013 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceb33c3aafdab4bea2de/log.txt.gz 2026-03-30 01:17:47,173 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:47,342 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:47,344 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceb33c3aafdab4bea2de/lava-logs.yaml 2026-03-30 01:17:47,351 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:47,511 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:47,513 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceb33c3aafdab4bea2de/lava_callback.json.gz 2026-03-30 01:17:47,536 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:47,550 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ceb33c3aafdab4bea2de HTTP/1.1" 200 None 2026-03-30 01:17:47,552 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:47,566 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:17:47,567 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:47,599 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ceb33c3aafdab4bea2de HTTP/1.1" 200 None 2026-03-30 01:17:47,600 - lava_callback - INFO - Completed processing callback for node 69c9ceb33c3aafdab4bea2de 2026-03-30 01:17:50,497 - lava_callback - INFO - Received callback for node 69c9cebf3c3aafdab4bea307 INFO: 172.20.0.1:41574 - "POST /node/69c9cebf3c3aafdab4bea307 HTTP/1.0" 202 Accepted 2026-03-30 01:17:50,504 - lava_callback - INFO - Starting async job submit for node 69c9cebf3c3aafdab4bea307 2026-03-30 01:17:50,524 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:50,538 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cebf3c3aafdab4bea307 HTTP/1.1" 200 None 2026-03-30 01:17:50,539 - lava_callback - DEBUG - Processing node 69c9cebf3c3aafdab4bea307: baseline-arm64-pengutronix 2026-03-30 01:17:50,740 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:50,856 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:50,857 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c9cebf3c3aafdab4bea307/log.txt.gz 2026-03-30 01:17:50,958 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:51,145 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:51,147 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c9cebf3c3aafdab4bea307/lava-logs.yaml 2026-03-30 01:17:51,152 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:17:51,299 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:17:51,301 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c9cebf3c3aafdab4bea307/lava_callback.json.gz 2026-03-30 01:17:51,343 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:51,371 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cebf3c3aafdab4bea307 HTTP/1.1" 200 None 2026-03-30 01:17:51,373 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:51,389 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:17:51,391 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:17:51,417 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9cebf3c3aafdab4bea307 HTTP/1.1" 200 None 2026-03-30 01:17:51,464 - lava_callback - INFO - Completed processing callback for node 69c9cebf3c3aafdab4bea307 INFO: 172.20.0.1:41582 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54784 - "GET /node/698a0e4d22b4cc0f227ad837%5Cn HTTP/1.0" 405 Method Not Allowed 2026-03-30 01:18:06,086 - lava_callback - INFO - Received callback for node 69c72c3929015d0f95992289 INFO: 172.20.0.1:54788 - "POST /node/69c72c3929015d0f95992289 HTTP/1.0" 202 Accepted 2026-03-30 01:18:06,092 - lava_callback - INFO - Starting async job submit for node 69c72c3929015d0f95992289 2026-03-30 01:18:06,093 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:18:06,107 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72c3929015d0f95992289 HTTP/1.1" 200 None 2026-03-30 01:18:06,108 - lava_callback - DEBUG - Processing node 69c72c3929015d0f95992289: baseline-arm-clabbe 2026-03-30 01:18:06,108 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72c3929015d0f95992289 2026-03-30 01:18:06,109 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:18:06,125 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72c3929015d0f95992289 HTTP/1.1" 200 None 2026-03-30 01:18:06,126 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:18:06,140 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69bf353af1004e82292e0db7 HTTP/1.1" 200 None 2026-03-30 01:18:06,142 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:18:06,162 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72c3929015d0f95992289 HTTP/1.1" 200 None 2026-03-30 01:18:06,163 - lava_callback - ERROR - Error processing callback for node 69c72c3929015d0f95992289: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54802 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54812 - "GET /node/69878b58e31b58f834f1bc29%5Cn HTTP/1.0" 405 Method Not Allowed 2026-03-30 01:18:11,806 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:18:11,827 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:48350 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48356 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48366 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59194 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56588 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:18:46,160 - lava_callback - INFO - Received callback for node 69c72c6029015d0f95992297 INFO: 172.20.0.1:51608 - "POST /node/69c72c6029015d0f95992297 HTTP/1.0" 202 Accepted 2026-03-30 01:18:46,166 - lava_callback - INFO - Starting async job submit for node 69c72c6029015d0f95992297 2026-03-30 01:18:46,167 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:18:46,195 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72c6029015d0f95992297 HTTP/1.1" 200 None 2026-03-30 01:18:46,196 - lava_callback - DEBUG - Processing node 69c72c6029015d0f95992297: baseline-arm-clabbe 2026-03-30 01:18:46,196 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72c6029015d0f95992297 2026-03-30 01:18:46,197 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:18:46,210 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72c6029015d0f95992297 HTTP/1.1" 200 None 2026-03-30 01:18:46,212 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:18:46,225 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69bf353af1004e82292e0db7 HTTP/1.1" 200 None 2026-03-30 01:18:46,227 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:18:46,253 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72c6029015d0f95992297 HTTP/1.1" 200 None 2026-03-30 01:18:46,256 - lava_callback - ERROR - Error processing callback for node 69c72c6029015d0f95992297: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:51614 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41676 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45678 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49592 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49608 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:19:17,627 - lava_callback - INFO - Received callback for node 69c9ceaf3c3aafdab4bea2d8 INFO: 172.20.0.1:49624 - "POST /node/69c9ceaf3c3aafdab4bea2d8 HTTP/1.0" 202 Accepted 2026-03-30 01:19:17,633 - lava_callback - INFO - Starting async job submit for node 69c9ceaf3c3aafdab4bea2d8 2026-03-30 01:19:17,655 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:19:17,670 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ceaf3c3aafdab4bea2d8 HTTP/1.1" 200 None 2026-03-30 01:19:17,671 - lava_callback - DEBUG - Processing node 69c9ceaf3c3aafdab4bea2d8: baseline-arm64-broonie 2026-03-30 01:19:17,969 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:19:18,114 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:19:18,116 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceaf3c3aafdab4bea2d8/log.txt.gz 2026-03-30 01:19:18,272 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.20.0.1:49636 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:19:18,637 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:19:18,639 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceaf3c3aafdab4bea2d8/lava-logs.yaml 2026-03-30 01:19:18,646 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:19:18,784 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:19:18,786 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceaf3c3aafdab4bea2d8/lava_callback.json.gz 2026-03-30 01:19:18,809 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:19:18,828 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ceaf3c3aafdab4bea2d8 HTTP/1.1" 200 None 2026-03-30 01:19:18,830 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:19:18,851 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:19:18,852 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:19:18,877 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ceaf3c3aafdab4bea2d8 HTTP/1.1" 200 None 2026-03-30 01:19:18,879 - lava_callback - INFO - Completed processing callback for node 69c9ceaf3c3aafdab4bea2d8 2026-03-30 01:19:26,163 - lava_callback - INFO - Received callback for node 69c72c8a29015d0f959922a9 INFO: 172.20.0.1:35496 - "POST /node/69c72c8a29015d0f959922a9 HTTP/1.0" 202 Accepted 2026-03-30 01:19:26,168 - lava_callback - INFO - Starting async job submit for node 69c72c8a29015d0f959922a9 2026-03-30 01:19:26,169 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:19:26,184 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72c8a29015d0f959922a9 HTTP/1.1" 200 None 2026-03-30 01:19:26,185 - lava_callback - DEBUG - Processing node 69c72c8a29015d0f959922a9: baseline-arm-clabbe 2026-03-30 01:19:26,185 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72c8a29015d0f959922a9 2026-03-30 01:19:26,186 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:19:26,198 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72c8a29015d0f959922a9 HTTP/1.1" 200 None 2026-03-30 01:19:26,200 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:19:26,213 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69bf353af1004e82292e0db7 HTTP/1.1" 200 None 2026-03-30 01:19:26,215 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:19:26,236 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72c8a29015d0f959922a9 HTTP/1.1" 200 None 2026-03-30 01:19:26,242 - lava_callback - ERROR - Error processing callback for node 69c72c8a29015d0f959922a9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35508 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45140 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:19:41,829 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:19:41,845 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:39102 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37272 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:19:57,674 - lava_callback - INFO - Received callback for node 69c9cea93c3aafdab4bea2b0 INFO: 172.20.0.1:37286 - "POST /node/69c9cea93c3aafdab4bea2b0 HTTP/1.0" 202 Accepted 2026-03-30 01:19:57,680 - lava_callback - INFO - Starting async job submit for node 69c9cea93c3aafdab4bea2b0 2026-03-30 01:19:57,702 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:19:57,717 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cea93c3aafdab4bea2b0 HTTP/1.1" 200 None 2026-03-30 01:19:57,717 - lava_callback - DEBUG - Processing node 69c9cea93c3aafdab4bea2b0: baseline-arm64-broonie 2026-03-30 01:19:57,985 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:19:58,133 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:19:58,135 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9cea93c3aafdab4bea2b0/log.txt.gz 2026-03-30 01:19:58,279 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:19:58,468 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:19:58,470 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9cea93c3aafdab4bea2b0/lava-logs.yaml 2026-03-30 01:19:58,477 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:19:58,621 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:19:58,623 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9cea93c3aafdab4bea2b0/lava_callback.json.gz 2026-03-30 01:19:58,646 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:19:58,671 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cea93c3aafdab4bea2b0 HTTP/1.1" 200 None 2026-03-30 01:19:58,672 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:19:58,686 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:19:58,688 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:19:58,713 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9cea93c3aafdab4bea2b0 HTTP/1.1" 200 None 2026-03-30 01:19:58,720 - lava_callback - INFO - Completed processing callback for node 69c9cea93c3aafdab4bea2b0 INFO: 172.20.0.1:50008 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:20:11,847 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:20:11,864 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-30 01:20:14,622 - lava_callback - INFO - Received callback for node 69c9cfc83c3aafdab4bea448 INFO: 172.20.0.1:44512 - "POST /node/69c9cfc83c3aafdab4bea448 HTTP/1.0" 202 Accepted 2026-03-30 01:20:14,628 - lava_callback - INFO - Starting async job submit for node 69c9cfc83c3aafdab4bea448 2026-03-30 01:20:14,630 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:20:14,645 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cfc83c3aafdab4bea448 HTTP/1.1" 200 None 2026-03-30 01:20:14,646 - lava_callback - DEBUG - Processing node 69c9cfc83c3aafdab4bea448: baseline-nfs-arm64 2026-03-30 01:20:14,647 - lava_callback - WARNING - Failed to upload log 2026-03-30 01:20:14,648 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:20:14,791 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:20:14,793 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9cfc83c3aafdab4bea448/lava-logs.yaml 2026-03-30 01:20:14,794 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:20:14,932 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:20:14,933 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9cfc83c3aafdab4bea448/lava_callback.json.gz 2026-03-30 01:20:14,935 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:20:14,950 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cfc83c3aafdab4bea448 HTTP/1.1" 200 None 2026-03-30 01:20:14,951 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:20:14,965 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:20:14,967 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:20:14,988 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9cfc83c3aafdab4bea448 HTTP/1.1" 200 None 2026-03-30 01:20:15,003 - lava_callback - INFO - Completed processing callback for node 69c9cfc83c3aafdab4bea448 INFO: 172.20.0.1:44518 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44532 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44542 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35880 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:20:27,967 - lava_callback - INFO - Received callback for node 69c9ce8e3c3aafdab4bea249 INFO: 172.20.0.1:35892 - "POST /node/69c9ce8e3c3aafdab4bea249 HTTP/1.0" 202 Accepted 2026-03-30 01:20:27,974 - lava_callback - INFO - Starting async job submit for node 69c9ce8e3c3aafdab4bea249 2026-03-30 01:20:28,010 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:20:28,024 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce8e3c3aafdab4bea249 HTTP/1.1" 200 None 2026-03-30 01:20:28,024 - lava_callback - DEBUG - Processing node 69c9ce8e3c3aafdab4bea249: baseline-arm64 2026-03-30 01:20:28,446 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:20:28,577 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:20:28,579 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce8e3c3aafdab4bea249/log.txt.gz 2026-03-30 01:20:28,797 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:20:29,002 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:20:29,004 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce8e3c3aafdab4bea249/lava-logs.yaml 2026-03-30 01:20:29,013 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:20:29,185 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:20:29,186 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce8e3c3aafdab4bea249/lava_callback.json.gz DEBUG: Setup failed for 69c9ce8e3c3aafdab4bea249. Transitting job node result: incomplete -> fail 2026-03-30 01:20:29,259 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:20:29,275 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce8e3c3aafdab4bea249 HTTP/1.1" 200 None 2026-03-30 01:20:29,277 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:20:29,290 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:20:29,291 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:20:29,330 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce8e3c3aafdab4bea249 HTTP/1.1" 200 None 2026-03-30 01:20:29,406 - lava_callback - INFO - Completed processing callback for node 69c9ce8e3c3aafdab4bea249 INFO: 172.20.0.1:34492 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:20:37,066 - lava_callback - INFO - Received callback for node 69c9cec23c3aafdab4bea30f INFO: 172.20.0.1:34500 - "POST /node/69c9cec23c3aafdab4bea30f HTTP/1.0" 202 Accepted 2026-03-30 01:20:37,072 - lava_callback - INFO - Starting async job submit for node 69c9cec23c3aafdab4bea30f 2026-03-30 01:20:37,092 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:20:37,112 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cec23c3aafdab4bea30f HTTP/1.1" 200 None 2026-03-30 01:20:37,113 - lava_callback - DEBUG - Processing node 69c9cec23c3aafdab4bea30f: baseline-arm64-pengutronix 2026-03-30 01:20:37,307 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:20:37,424 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:20:37,426 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c9cec23c3aafdab4bea30f/log.txt.gz 2026-03-30 01:20:37,547 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:20:37,802 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:20:37,804 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c9cec23c3aafdab4bea30f/lava-logs.yaml 2026-03-30 01:20:37,810 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:20:37,954 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:20:37,956 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c9cec23c3aafdab4bea30f/lava_callback.json.gz 2026-03-30 01:20:37,976 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:20:37,991 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cec23c3aafdab4bea30f HTTP/1.1" 200 None 2026-03-30 01:20:37,992 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:20:38,008 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:20:38,009 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:20:38,036 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9cec23c3aafdab4bea30f HTTP/1.1" 200 None 2026-03-30 01:20:38,041 - lava_callback - INFO - Completed processing callback for node 69c9cec23c3aafdab4bea30f 2026-03-30 01:20:41,867 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:20:41,883 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:33086 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51296 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:20:59,191 - lava_callback - INFO - Received callback for node 69c9ceaa3c3aafdab4bea2b3 INFO: 172.20.0.1:51304 - "POST /node/69c9ceaa3c3aafdab4bea2b3 HTTP/1.0" 202 Accepted 2026-03-30 01:20:59,197 - lava_callback - INFO - Starting async job submit for node 69c9ceaa3c3aafdab4bea2b3 2026-03-30 01:20:59,219 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:20:59,254 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ceaa3c3aafdab4bea2b3 HTTP/1.1" 200 None 2026-03-30 01:20:59,255 - lava_callback - DEBUG - Processing node 69c9ceaa3c3aafdab4bea2b3: baseline-arm64-broonie 2026-03-30 01:20:59,484 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:20:59,603 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:20:59,604 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceaa3c3aafdab4bea2b3/log.txt.gz 2026-03-30 01:20:59,755 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:20:59,981 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:20:59,983 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceaa3c3aafdab4bea2b3/lava-logs.yaml 2026-03-30 01:20:59,989 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:21:00,141 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:21:00,143 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69c9ceaa3c3aafdab4bea2b3/lava_callback.json.gz 2026-03-30 01:21:00,166 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:00,181 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ceaa3c3aafdab4bea2b3 HTTP/1.1" 200 None 2026-03-30 01:21:00,183 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:00,198 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:21:00,200 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:00,224 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ceaa3c3aafdab4bea2b3 HTTP/1.1" 200 None 2026-03-30 01:21:00,226 - lava_callback - INFO - Completed processing callback for node 69c9ceaa3c3aafdab4bea2b3 2026-03-30 01:21:04,342 - lava_callback - INFO - Received callback for node 69c72cf929015d0f9599235f INFO: 172.20.0.1:47648 - "POST /node/69c72cf929015d0f9599235f HTTP/1.0" 202 Accepted 2026-03-30 01:21:04,348 - lava_callback - INFO - Starting async job submit for node 69c72cf929015d0f9599235f 2026-03-30 01:21:04,349 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:04,374 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72cf929015d0f9599235f HTTP/1.1" 200 None 2026-03-30 01:21:04,375 - lava_callback - DEBUG - Processing node 69c72cf929015d0f9599235f: baseline-arm64-baylibre 2026-03-30 01:21:04,375 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72cf929015d0f9599235f 2026-03-30 01:21:04,376 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:04,392 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72cf929015d0f9599235f HTTP/1.1" 200 None 2026-03-30 01:21:04,393 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:04,436 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 01:21:04,438 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:04,469 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72cf929015d0f9599235f HTTP/1.1" 200 None 2026-03-30 01:21:04,471 - lava_callback - ERROR - Error processing callback for node 69c72cf929015d0f9599235f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-30 01:21:05,008 - lava_callback - INFO - Received callback for node 69c72cfb29015d0f9599236a INFO: 172.20.0.1:47656 - "POST /node/69c72cfb29015d0f9599236a HTTP/1.0" 202 Accepted 2026-03-30 01:21:05,014 - lava_callback - INFO - Starting async job submit for node 69c72cfb29015d0f9599236a 2026-03-30 01:21:05,015 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:05,030 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72cfb29015d0f9599236a HTTP/1.1" 200 None 2026-03-30 01:21:05,031 - lava_callback - DEBUG - Processing node 69c72cfb29015d0f9599236a: baseline-arm64-baylibre 2026-03-30 01:21:05,031 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72cfb29015d0f9599236a 2026-03-30 01:21:05,032 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:05,047 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72cfb29015d0f9599236a HTTP/1.1" 200 None 2026-03-30 01:21:05,049 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:05,063 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 01:21:05,064 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:05,085 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72cfb29015d0f9599236a HTTP/1.1" 200 None 2026-03-30 01:21:05,086 - lava_callback - ERROR - Error processing callback for node 69c72cfb29015d0f9599236a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-30 01:21:05,690 - lava_callback - INFO - Received callback for node 69c72cf829015d0f9599235c INFO: 172.20.0.1:47672 - "POST /node/69c72cf829015d0f9599235c HTTP/1.0" 202 Accepted 2026-03-30 01:21:05,696 - lava_callback - INFO - Starting async job submit for node 69c72cf829015d0f9599235c 2026-03-30 01:21:05,697 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:05,711 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72cf829015d0f9599235c HTTP/1.1" 200 None 2026-03-30 01:21:05,711 - lava_callback - DEBUG - Processing node 69c72cf829015d0f9599235c: baseline-arm64-baylibre 2026-03-30 01:21:05,712 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72cf829015d0f9599235c 2026-03-30 01:21:05,712 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:05,737 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72cf829015d0f9599235c HTTP/1.1" 200 None 2026-03-30 01:21:05,739 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:05,752 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 01:21:05,754 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:05,774 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72cf829015d0f9599235c HTTP/1.1" 200 None 2026-03-30 01:21:05,778 - lava_callback - ERROR - Error processing callback for node 69c72cf829015d0f9599235c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-30 01:21:06,367 - lava_callback - INFO - Received callback for node 69c72cfd29015d0f95992373 INFO: 172.20.0.1:47678 - "POST /node/69c72cfd29015d0f95992373 HTTP/1.0" 202 Accepted 2026-03-30 01:21:06,372 - lava_callback - INFO - Starting async job submit for node 69c72cfd29015d0f95992373 2026-03-30 01:21:06,373 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:06,389 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72cfd29015d0f95992373 HTTP/1.1" 200 None 2026-03-30 01:21:06,390 - lava_callback - DEBUG - Processing node 69c72cfd29015d0f95992373: baseline-arm64-baylibre 2026-03-30 01:21:06,390 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72cfd29015d0f95992373 2026-03-30 01:21:06,391 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:06,404 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72cfd29015d0f95992373 HTTP/1.1" 200 None 2026-03-30 01:21:06,406 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:06,419 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 01:21:06,421 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:06,441 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72cfd29015d0f95992373 HTTP/1.1" 200 None 2026-03-30 01:21:06,442 - lava_callback - ERROR - Error processing callback for node 69c72cfd29015d0f95992373: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47694 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:21:07,049 - lava_callback - INFO - Received callback for node 69c72cfe29015d0f95992376 INFO: 172.20.0.1:47700 - "POST /node/69c72cfe29015d0f95992376 HTTP/1.0" 202 Accepted 2026-03-30 01:21:07,055 - lava_callback - INFO - Starting async job submit for node 69c72cfe29015d0f95992376 2026-03-30 01:21:07,056 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:07,072 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72cfe29015d0f95992376 HTTP/1.1" 200 None 2026-03-30 01:21:07,073 - lava_callback - DEBUG - Processing node 69c72cfe29015d0f95992376: baseline-arm64-baylibre 2026-03-30 01:21:07,073 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72cfe29015d0f95992376 2026-03-30 01:21:07,074 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:07,089 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72cfe29015d0f95992376 HTTP/1.1" 200 None 2026-03-30 01:21:07,090 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:07,104 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 01:21:07,105 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:07,126 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72cfe29015d0f95992376 HTTP/1.1" 200 None 2026-03-30 01:21:07,132 - lava_callback - ERROR - Error processing callback for node 69c72cfe29015d0f95992376: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-30 01:21:11,885 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:11,901 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:54850 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54860 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54870 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:21:24,246 - lava_callback - INFO - Received callback for node 69c72d0029015d0f95992379 INFO: 172.20.0.1:46242 - "POST /node/69c72d0029015d0f95992379 HTTP/1.0" 202 Accepted 2026-03-30 01:21:24,252 - lava_callback - INFO - Starting async job submit for node 69c72d0029015d0f95992379 2026-03-30 01:21:24,253 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:24,269 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72d0029015d0f95992379 HTTP/1.1" 200 None 2026-03-30 01:21:24,269 - lava_callback - DEBUG - Processing node 69c72d0029015d0f95992379: baseline-arm64-baylibre 2026-03-30 01:21:24,270 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72d0029015d0f95992379 2026-03-30 01:21:24,270 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:24,284 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72d0029015d0f95992379 HTTP/1.1" 200 None 2026-03-30 01:21:24,285 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:24,302 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 01:21:24,304 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:24,325 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72d0029015d0f95992379 HTTP/1.1" 200 None 2026-03-30 01:21:24,327 - lava_callback - ERROR - Error processing callback for node 69c72d0029015d0f95992379: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-30 01:21:26,281 - lava_callback - INFO - Received callback for node 69c72d1029015d0f959923c2 INFO: 172.20.0.1:46246 - "POST /node/69c72d1029015d0f959923c2 HTTP/1.0" 202 Accepted 2026-03-30 01:21:26,287 - lava_callback - INFO - Starting async job submit for node 69c72d1029015d0f959923c2 2026-03-30 01:21:26,287 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:26,305 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72d1029015d0f959923c2 HTTP/1.1" 200 None 2026-03-30 01:21:26,306 - lava_callback - DEBUG - Processing node 69c72d1029015d0f959923c2: baseline-arm64-clabbe 2026-03-30 01:21:26,306 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72d1029015d0f959923c2 2026-03-30 01:21:26,307 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:26,321 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72d1029015d0f959923c2 HTTP/1.1" 200 None 2026-03-30 01:21:26,322 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:26,337 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 01:21:26,339 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:26,359 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72d1029015d0f959923c2 HTTP/1.1" 200 None 2026-03-30 01:21:26,365 - lava_callback - ERROR - Error processing callback for node 69c72d1029015d0f959923c2: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46262 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:21:26,955 - lava_callback - INFO - Received callback for node 69c72d1229015d0f959923c5 INFO: 172.20.0.1:46264 - "POST /node/69c72d1229015d0f959923c5 HTTP/1.0" 202 Accepted 2026-03-30 01:21:26,960 - lava_callback - INFO - Starting async job submit for node 69c72d1229015d0f959923c5 2026-03-30 01:21:26,961 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:26,979 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72d1229015d0f959923c5 HTTP/1.1" 200 None 2026-03-30 01:21:26,980 - lava_callback - DEBUG - Processing node 69c72d1229015d0f959923c5: baseline-arm64-clabbe 2026-03-30 01:21:26,980 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72d1229015d0f959923c5 2026-03-30 01:21:26,981 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:26,998 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72d1229015d0f959923c5 HTTP/1.1" 200 None 2026-03-30 01:21:26,999 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:27,013 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 01:21:27,014 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:27,038 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72d1229015d0f959923c5 HTTP/1.1" 200 None 2026-03-30 01:21:27,039 - lava_callback - ERROR - Error processing callback for node 69c72d1229015d0f959923c5: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-30 01:21:27,628 - lava_callback - INFO - Received callback for node 69c72d1329015d0f959923c8 INFO: 172.20.0.1:46266 - "POST /node/69c72d1329015d0f959923c8 HTTP/1.0" 202 Accepted 2026-03-30 01:21:27,637 - lava_callback - INFO - Starting async job submit for node 69c72d1329015d0f959923c8 2026-03-30 01:21:27,638 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:27,652 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72d1329015d0f959923c8 HTTP/1.1" 200 None 2026-03-30 01:21:27,653 - lava_callback - DEBUG - Processing node 69c72d1329015d0f959923c8: baseline-arm64-clabbe 2026-03-30 01:21:27,653 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72d1329015d0f959923c8 2026-03-30 01:21:27,654 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:27,676 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72d1329015d0f959923c8 HTTP/1.1" 200 None 2026-03-30 01:21:27,677 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:27,690 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 01:21:27,692 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:27,721 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72d1329015d0f959923c8 HTTP/1.1" 200 None 2026-03-30 01:21:27,736 - lava_callback - ERROR - Error processing callback for node 69c72d1329015d0f959923c8: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-30 01:21:28,320 - lava_callback - INFO - Received callback for node 69c72d1329015d0f959923cb INFO: 172.20.0.1:46278 - "POST /node/69c72d1329015d0f959923cb HTTP/1.0" 202 Accepted 2026-03-30 01:21:28,326 - lava_callback - INFO - Starting async job submit for node 69c72d1329015d0f959923cb 2026-03-30 01:21:28,326 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:28,343 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72d1329015d0f959923cb HTTP/1.1" 200 None 2026-03-30 01:21:28,345 - lava_callback - DEBUG - Processing node 69c72d1329015d0f959923cb: baseline-arm64-clabbe 2026-03-30 01:21:28,345 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72d1329015d0f959923cb 2026-03-30 01:21:28,346 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:28,365 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72d1329015d0f959923cb HTTP/1.1" 200 None 2026-03-30 01:21:28,367 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:28,380 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 01:21:28,382 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:28,404 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72d1329015d0f959923cb HTTP/1.1" 200 None 2026-03-30 01:21:28,410 - lava_callback - ERROR - Error processing callback for node 69c72d1329015d0f959923cb: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:56760 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:21:46,249 - lava_callback - INFO - Received callback for node 69c72d1629015d0f959923d3 INFO: 172.20.0.1:32956 - "POST /node/69c72d1629015d0f959923d3 HTTP/1.0" 202 Accepted 2026-03-30 01:21:46,255 - lava_callback - INFO - Starting async job submit for node 69c72d1629015d0f959923d3 2026-03-30 01:21:46,256 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:46,273 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72d1629015d0f959923d3 HTTP/1.1" 200 None 2026-03-30 01:21:46,273 - lava_callback - DEBUG - Processing node 69c72d1629015d0f959923d3: baseline-arm64-clabbe 2026-03-30 01:21:46,274 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72d1629015d0f959923d3 2026-03-30 01:21:46,274 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:46,289 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72d1629015d0f959923d3 HTTP/1.1" 200 None 2026-03-30 01:21:46,290 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:46,305 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 01:21:46,307 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:46,327 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72d1629015d0f959923d3 HTTP/1.1" 200 None 2026-03-30 01:21:46,329 - lava_callback - ERROR - Error processing callback for node 69c72d1629015d0f959923d3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:32970 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:21:46,948 - lava_callback - INFO - Received callback for node 69c72d1629015d0f959923d6 INFO: 172.20.0.1:32986 - "POST /node/69c72d1629015d0f959923d6 HTTP/1.0" 202 Accepted 2026-03-30 01:21:46,957 - lava_callback - INFO - Starting async job submit for node 69c72d1629015d0f959923d6 2026-03-30 01:21:46,958 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:46,973 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72d1629015d0f959923d6 HTTP/1.1" 200 None 2026-03-30 01:21:46,974 - lava_callback - DEBUG - Processing node 69c72d1629015d0f959923d6: baseline-arm64-clabbe 2026-03-30 01:21:46,974 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72d1629015d0f959923d6 2026-03-30 01:21:46,975 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:46,996 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72d1629015d0f959923d6 HTTP/1.1" 200 None 2026-03-30 01:21:46,998 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:47,016 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 01:21:47,018 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:21:47,038 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72d1629015d0f959923d6 HTTP/1.1" 200 None 2026-03-30 01:21:47,040 - lava_callback - ERROR - Error processing callback for node 69c72d1629015d0f959923d6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:34020 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:22:04,312 - lava_callback - INFO - Received callback for node 69c72d2a29015d0f95992413 INFO: 172.20.0.1:55860 - "POST /node/69c72d2a29015d0f95992413 HTTP/1.0" 202 Accepted 2026-03-30 01:22:04,317 - lava_callback - INFO - Starting async job submit for node 69c72d2a29015d0f95992413 2026-03-30 01:22:04,318 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:22:04,332 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72d2a29015d0f95992413 HTTP/1.1" 200 None 2026-03-30 01:22:04,333 - lava_callback - DEBUG - Processing node 69c72d2a29015d0f95992413: baseline-arm64-baylibre 2026-03-30 01:22:04,334 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72d2a29015d0f95992413 2026-03-30 01:22:04,334 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:22:04,347 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72d2a29015d0f95992413 HTTP/1.1" 200 None 2026-03-30 01:22:04,348 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:22:04,362 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 01:22:04,363 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:22:04,407 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72d2a29015d0f95992413 HTTP/1.1" 200 None 2026-03-30 01:22:04,409 - lava_callback - ERROR - Error processing callback for node 69c72d2a29015d0f95992413: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55864 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40796 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40800 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40816 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34318 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:22:27,391 - lava_callback - INFO - Received callback for node 69c9d00e3c3aafdab4bea473 INFO: 172.20.0.1:34324 - "POST /node/69c9d00e3c3aafdab4bea473 HTTP/1.0" 202 Accepted 2026-03-30 01:22:27,398 - lava_callback - INFO - Starting async job submit for node 69c9d00e3c3aafdab4bea473 2026-03-30 01:22:27,428 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:22:27,443 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d00e3c3aafdab4bea473 HTTP/1.1" 200 None 2026-03-30 01:22:27,444 - lava_callback - DEBUG - Processing node 69c9d00e3c3aafdab4bea473: baseline-arm64 2026-03-30 01:22:27,580 - lava_callback - WARNING - Failed to upload log 2026-03-30 01:22:27,651 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:22:27,836 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:22:27,837 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d00e3c3aafdab4bea473/lava-logs.yaml 2026-03-30 01:22:27,841 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:22:28,129 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:22:28,131 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d00e3c3aafdab4bea473/lava_callback.json.gz 2026-03-30 01:22:28,161 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:22:28,175 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d00e3c3aafdab4bea473 HTTP/1.1" 200 None 2026-03-30 01:22:28,177 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:22:28,190 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:22:28,192 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:22:28,213 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9d00e3c3aafdab4bea473 HTTP/1.1" 200 None 2026-03-30 01:22:28,299 - lava_callback - INFO - Completed processing callback for node 69c9d00e3c3aafdab4bea473 INFO: 172.20.0.1:60894 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:22:41,903 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:22:41,944 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:57712 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45166 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47960 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57390 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57406 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57422 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38222 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57510 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:23:40,905 - lava_callback - INFO - Received callback for node 69c9d0613c3aafdab4bea48d INFO: 172.20.0.1:58832 - "POST /node/69c9d0613c3aafdab4bea48d HTTP/1.0" 202 Accepted 2026-03-30 01:23:40,912 - lava_callback - INFO - Starting async job submit for node 69c9d0613c3aafdab4bea48d 2026-03-30 01:23:40,944 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:23:40,959 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d0613c3aafdab4bea48d HTTP/1.1" 200 None 2026-03-30 01:23:40,960 - lava_callback - DEBUG - Processing node 69c9d0613c3aafdab4bea48d: baseline-arm64 2026-03-30 01:23:41,146 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:23:41,473 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:23:41,476 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d0613c3aafdab4bea48d/log.txt.gz 2026-03-30 01:23:41,563 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:23:41,733 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:23:41,734 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d0613c3aafdab4bea48d/lava-logs.yaml 2026-03-30 01:23:41,738 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:23:42,113 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:23:42,115 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d0613c3aafdab4bea48d/lava_callback.json.gz 2026-03-30 01:23:42,169 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:23:42,216 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d0613c3aafdab4bea48d HTTP/1.1" 200 None 2026-03-30 01:23:42,219 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:23:42,244 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:23:42,247 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:23:42,281 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9d0613c3aafdab4bea48d HTTP/1.1" 200 None 2026-03-30 01:23:42,392 - lava_callback - INFO - Completed processing callback for node 69c9d0613c3aafdab4bea48d INFO: 172.20.0.1:58848 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56506 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48330 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:24:11,947 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:24:11,973 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:36510 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36518 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36534 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36978 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39884 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:24:38,292 - lava_callback - INFO - Received callback for node 69c9ccbb3c3aafdab4bea17c INFO: 172.20.0.1:39898 - "POST /node/69c9ccbb3c3aafdab4bea17c HTTP/1.0" 202 Accepted 2026-03-30 01:24:38,299 - lava_callback - INFO - Starting async job submit for node 69c9ccbb3c3aafdab4bea17c 2026-03-30 01:24:38,359 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:24:38,374 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ccbb3c3aafdab4bea17c HTTP/1.1" 200 None 2026-03-30 01:24:38,375 - lava_callback - DEBUG - Processing node 69c9ccbb3c3aafdab4bea17c: ltp-smoke 2026-03-30 01:24:38,880 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:24:39,002 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:24:39,004 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69c9ccbb3c3aafdab4bea17c/log.txt.gz 2026-03-30 01:24:39,287 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:24:39,522 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:24:39,524 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69c9ccbb3c3aafdab4bea17c/lava-logs.yaml 2026-03-30 01:24:39,536 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:24:39,682 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:24:39,683 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69c9ccbb3c3aafdab4bea17c/lava_callback.json.gz 2026-03-30 01:24:39,810 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:24:39,824 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ccbb3c3aafdab4bea17c HTTP/1.1" 200 None 2026-03-30 01:24:39,826 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:24:39,844 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1213c3aafdab4be774c HTTP/1.1" 200 None 2026-03-30 01:24:39,845 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:24:39,870 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ccbb3c3aafdab4bea17c HTTP/1.1" 200 None 2026-03-30 01:24:40,003 - lava_callback - INFO - Completed processing callback for node 69c9ccbb3c3aafdab4bea17c 2026-03-30 01:24:41,976 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:24:41,993 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:50600 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60782 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53676 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35756 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35768 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35770 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58738 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:25:31,092 - lava_callback - INFO - Received callback for node 69c9d0b33c3aafdab4bea499 INFO: 172.20.0.1:60640 - "POST /node/69c9d0b33c3aafdab4bea499 HTTP/1.0" 202 Accepted 2026-03-30 01:25:31,099 - lava_callback - INFO - Starting async job submit for node 69c9d0b33c3aafdab4bea499 2026-03-30 01:25:31,121 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:25:31,156 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d0b33c3aafdab4bea499 HTTP/1.1" 200 None 2026-03-30 01:25:31,157 - lava_callback - DEBUG - Processing node 69c9d0b33c3aafdab4bea499: baseline-arm64 2026-03-30 01:25:31,597 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:25:31,787 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:25:31,789 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d0b33c3aafdab4bea499/log.txt.gz 2026-03-30 01:25:32,043 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:25:32,343 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:25:32,345 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d0b33c3aafdab4bea499/lava-logs.yaml 2026-03-30 01:25:32,355 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:25:32,507 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:25:32,509 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d0b33c3aafdab4bea499/lava_callback.json.gz 2026-03-30 01:25:32,552 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:25:32,568 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d0b33c3aafdab4bea499 HTTP/1.1" 200 None 2026-03-30 01:25:32,569 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:25:32,588 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:25:32,590 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:25:32,615 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9d0b33c3aafdab4bea499 HTTP/1.1" 200 None 2026-03-30 01:25:32,669 - lava_callback - INFO - Completed processing callback for node 69c9d0b33c3aafdab4bea499 INFO: 172.20.0.1:60650 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:25:41,995 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:25:42,010 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:59964 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33206 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:25:58,063 - lava_callback - INFO - Received callback for node 69c9cf6c3c3aafdab4bea42c INFO: 172.20.0.1:33222 - "POST /node/69c9cf6c3c3aafdab4bea42c HTTP/1.0" 202 Accepted 2026-03-30 01:25:58,070 - lava_callback - INFO - Starting async job submit for node 69c9cf6c3c3aafdab4bea42c 2026-03-30 01:25:58,106 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:25:58,125 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cf6c3c3aafdab4bea42c HTTP/1.1" 200 None 2026-03-30 01:25:58,126 - lava_callback - DEBUG - Processing node 69c9cf6c3c3aafdab4bea42c: ltp-ipc 2026-03-30 01:25:58,525 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:25:58,687 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:25:58,689 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-ipc-69c9cf6c3c3aafdab4bea42c/log.txt.gz 2026-03-30 01:25:58,950 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:25:59,177 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:25:59,179 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-ipc-69c9cf6c3c3aafdab4bea42c/lava-logs.yaml 2026-03-30 01:25:59,188 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:25:59,528 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:25:59,530 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-ipc-69c9cf6c3c3aafdab4bea42c/lava_callback.json.gz 2026-03-30 01:25:59,566 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:25:59,581 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cf6c3c3aafdab4bea42c HTTP/1.1" 200 None 2026-03-30 01:25:59,583 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:25:59,601 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 01:25:59,602 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:25:59,627 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9cf6c3c3aafdab4bea42c HTTP/1.1" 200 None 2026-03-30 01:25:59,628 - lava_callback - INFO - Completed processing callback for node 69c9cf6c3c3aafdab4bea42c INFO: 172.20.0.1:59612 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:26:12,012 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:26:12,029 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:39342 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39356 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39372 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40420 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41714 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40966 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46358 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50522 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58338 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58350 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58362 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50680 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51058 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:27:44,034 - lava_callback - INFO - Received callback for node 69c9d1173c3aafdab4bea4b7 INFO: 172.20.0.1:37360 - "POST /node/69c9d1173c3aafdab4bea4b7 HTTP/1.0" 202 Accepted 2026-03-30 01:27:44,041 - lava_callback - INFO - Starting async job submit for node 69c9d1173c3aafdab4bea4b7 2026-03-30 01:27:44,062 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:27:44,076 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d1173c3aafdab4bea4b7 HTTP/1.1" 200 None 2026-03-30 01:27:44,077 - lava_callback - DEBUG - Processing node 69c9d1173c3aafdab4bea4b7: baseline-arm64-pengutronix 2026-03-30 01:27:44,248 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:27:44,394 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:27:44,396 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c9d1173c3aafdab4bea4b7/log.txt.gz 2026-03-30 01:27:44,510 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:27:44,729 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:27:44,731 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c9d1173c3aafdab4bea4b7/lava-logs.yaml 2026-03-30 01:27:44,736 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:27:44,885 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:27:44,886 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c9d1173c3aafdab4bea4b7/lava_callback.json.gz 2026-03-30 01:27:44,928 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:27:44,943 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d1173c3aafdab4bea4b7 HTTP/1.1" 200 None 2026-03-30 01:27:44,945 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:27:44,960 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:27:44,962 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:27:44,987 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9d1173c3aafdab4bea4b7 HTTP/1.1" 200 None 2026-03-30 01:27:45,044 - lava_callback - INFO - Completed processing callback for node 69c9d1173c3aafdab4bea4b7 INFO: 172.20.0.1:37376 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55798 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52214 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:28:12,031 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:28:12,046 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:49364 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49368 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:28:17,533 - lava_callback - INFO - Received callback for node 69c9d1a53c3aafdab4bea4d3 INFO: 172.20.0.1:49372 - "POST /node/69c9d1a53c3aafdab4bea4d3 HTTP/1.0" 202 Accepted 2026-03-30 01:28:17,539 - lava_callback - INFO - Starting async job submit for node 69c9d1a53c3aafdab4bea4d3 2026-03-30 01:28:17,541 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:28:17,554 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d1a53c3aafdab4bea4d3 HTTP/1.1" 200 None 2026-03-30 01:28:17,555 - lava_callback - DEBUG - Processing node 69c9d1a53c3aafdab4bea4d3: baseline-nfs-arm64 2026-03-30 01:28:17,557 - lava_callback - WARNING - Failed to upload log 2026-03-30 01:28:17,558 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:28:17,704 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:28:17,706 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9d1a53c3aafdab4bea4d3/lava-logs.yaml 2026-03-30 01:28:17,707 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:28:17,853 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:28:17,855 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9d1a53c3aafdab4bea4d3/lava_callback.json.gz 2026-03-30 01:28:17,857 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:28:17,872 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d1a53c3aafdab4bea4d3 HTTP/1.1" 200 None 2026-03-30 01:28:17,873 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:28:17,887 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:28:17,889 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:28:17,922 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9d1a53c3aafdab4bea4d3 HTTP/1.1" 200 None 2026-03-30 01:28:17,925 - lava_callback - INFO - Completed processing callback for node 69c9d1a53c3aafdab4bea4d3 INFO: 172.20.0.1:49384 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58028 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46962 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:28:42,048 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:28:42,064 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:41800 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35056 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:29:04,819 - lava_callback - INFO - Received callback for node 69c72ed829015d0f9599254e INFO: 172.20.0.1:48982 - "POST /node/69c72ed829015d0f9599254e HTTP/1.0" 202 Accepted 2026-03-30 01:29:04,824 - lava_callback - INFO - Starting async job submit for node 69c72ed829015d0f9599254e 2026-03-30 01:29:04,825 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:29:04,839 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72ed829015d0f9599254e HTTP/1.1" 200 None 2026-03-30 01:29:04,840 - lava_callback - DEBUG - Processing node 69c72ed829015d0f9599254e: baseline-arm64-baylibre 2026-03-30 01:29:04,840 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72ed829015d0f9599254e 2026-03-30 01:29:04,841 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:29:04,854 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72ed829015d0f9599254e HTTP/1.1" 200 None 2026-03-30 01:29:04,855 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:29:04,869 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 01:29:04,870 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:29:04,892 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72ed829015d0f9599254e HTTP/1.1" 200 None 2026-03-30 01:29:04,900 - lava_callback - ERROR - Error processing callback for node 69c72ed829015d0f9599254e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48996 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42532 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42548 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43684 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45726 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44682 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59732 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52804 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54520 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54536 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54548 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56104 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52076 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45546 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48196 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:31:05,344 - lava_callback - INFO - Received callback for node 69c72f5129015d0f95992568 INFO: 172.20.0.1:54896 - "POST /node/69c72f5129015d0f95992568 HTTP/1.0" 202 Accepted 2026-03-30 01:31:05,350 - lava_callback - INFO - Starting async job submit for node 69c72f5129015d0f95992568 2026-03-30 01:31:05,351 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:31:05,365 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72f5129015d0f95992568 HTTP/1.1" 200 None 2026-03-30 01:31:05,366 - lava_callback - DEBUG - Processing node 69c72f5129015d0f95992568: baseline-arm64-baylibre 2026-03-30 01:31:05,366 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72f5129015d0f95992568 2026-03-30 01:31:05,367 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:31:05,380 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72f5129015d0f95992568 HTTP/1.1" 200 None 2026-03-30 01:31:05,381 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:31:05,394 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 01:31:05,396 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:31:05,420 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72f5129015d0f95992568 HTTP/1.1" 200 None 2026-03-30 01:31:05,425 - lava_callback - ERROR - Error processing callback for node 69c72f5129015d0f95992568: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54910 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49836 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49852 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49862 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34538 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36586 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51104 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54814 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38224 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60648 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60654 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:32:24,983 - lava_callback - INFO - Received callback for node 69c72fa329015d0f9599258e INFO: 172.20.0.1:33516 - "POST /node/69c72fa329015d0f9599258e HTTP/1.0" 202 Accepted 2026-03-30 01:32:24,989 - lava_callback - INFO - Starting async job submit for node 69c72fa329015d0f9599258e 2026-03-30 01:32:24,989 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:32:25,005 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72fa329015d0f9599258e HTTP/1.1" 200 None 2026-03-30 01:32:25,006 - lava_callback - DEBUG - Processing node 69c72fa329015d0f9599258e: baseline-arm64-baylibre 2026-03-30 01:32:25,007 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72fa329015d0f9599258e 2026-03-30 01:32:25,007 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:32:25,020 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72fa329015d0f9599258e HTTP/1.1" 200 None 2026-03-30 01:32:25,022 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:32:25,035 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 01:32:25,036 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:32:25,059 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72fa329015d0f9599258e HTTP/1.1" 200 None 2026-03-30 01:32:25,085 - lava_callback - ERROR - Error processing callback for node 69c72fa329015d0f9599258e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:33532 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:32:32,233 - lava_callback - INFO - Received callback for node 69c9ce713c3aafdab4bea202 INFO: 172.20.0.1:44216 - "POST /node/69c9ce713c3aafdab4bea202 HTTP/1.0" 202 Accepted 2026-03-30 01:32:32,240 - lava_callback - INFO - Starting async job submit for node 69c9ce713c3aafdab4bea202 2026-03-30 01:32:32,303 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:32:32,328 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce713c3aafdab4bea202 HTTP/1.1" 200 None 2026-03-30 01:32:32,329 - lava_callback - DEBUG - Processing node 69c9ce713c3aafdab4bea202: baseline-arm 2026-03-30 01:32:32,954 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:32:33,255 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:32:33,257 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69c9ce713c3aafdab4bea202/log.txt.gz 2026-03-30 01:32:33,612 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:32:33,871 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:32:33,873 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c9ce713c3aafdab4bea202/lava-logs.yaml 2026-03-30 01:32:33,888 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:32:34,067 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:32:34,069 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c9ce713c3aafdab4bea202/lava_callback.json.gz DEBUG: Setup failed for 69c9ce713c3aafdab4bea202. Transitting job node result: incomplete -> fail 2026-03-30 01:32:34,178 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:32:34,192 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce713c3aafdab4bea202 HTTP/1.1" 200 None 2026-03-30 01:32:34,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:32:34,207 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 01:32:34,208 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:32:34,251 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce713c3aafdab4bea202 HTTP/1.1" 200 None 2026-03-30 01:32:34,382 - lava_callback - INFO - Completed processing callback for node 69c9ce713c3aafdab4bea202 INFO: 172.20.0.1:44222 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:32:42,066 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:32:42,092 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:48632 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50058 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32874 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33540 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33554 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33560 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43100 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47388 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:33:44,969 - lava_callback - INFO - Received callback for node 69c72ff729015d0f959925a3 INFO: 172.20.0.1:60428 - "POST /node/69c72ff729015d0f959925a3 HTTP/1.0" 202 Accepted 2026-03-30 01:33:44,975 - lava_callback - INFO - Starting async job submit for node 69c72ff729015d0f959925a3 2026-03-30 01:33:44,976 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:33:44,999 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72ff729015d0f959925a3 HTTP/1.1" 200 None 2026-03-30 01:33:45,000 - lava_callback - DEBUG - Processing node 69c72ff729015d0f959925a3: baseline-arm64-baylibre 2026-03-30 01:33:45,000 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c72ff729015d0f959925a3 2026-03-30 01:33:45,001 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:33:45,019 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c72ff729015d0f959925a3 HTTP/1.1" 200 None 2026-03-30 01:33:45,021 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:33:45,036 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b4a590270ad7b6d699d HTTP/1.1" 200 None 2026-03-30 01:33:45,038 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:33:45,092 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c72ff729015d0f959925a3 HTTP/1.1" 200 None 2026-03-30 01:33:45,094 - lava_callback - ERROR - Error processing callback for node 69c72ff729015d0f959925a3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:60444 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40456 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50960 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53604 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53608 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53622 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33680 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40310 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:34:42,405 - lava_callback - INFO - Received callback for node 69c9d1f13c3aafdab4bea4e1 INFO: 172.20.0.1:60654 - "POST /node/69c9d1f13c3aafdab4bea4e1 HTTP/1.0" 202 Accepted 2026-03-30 01:34:42,412 - lava_callback - INFO - Starting async job submit for node 69c9d1f13c3aafdab4bea4e1 2026-03-30 01:34:42,448 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:34:42,463 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d1f13c3aafdab4bea4e1 HTTP/1.1" 200 None 2026-03-30 01:34:42,464 - lava_callback - DEBUG - Processing node 69c9d1f13c3aafdab4bea4e1: baseline-arm64 2026-03-30 01:34:42,858 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:34:43,016 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:34:43,017 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d1f13c3aafdab4bea4e1/log.txt.gz 2026-03-30 01:34:43,223 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:34:43,465 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:34:43,467 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d1f13c3aafdab4bea4e1/lava-logs.yaml 2026-03-30 01:34:43,476 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:34:43,642 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:34:43,643 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d1f13c3aafdab4bea4e1/lava_callback.json.gz DEBUG: Setup failed for 69c9d1f13c3aafdab4bea4e1. Transitting job node result: incomplete -> fail 2026-03-30 01:34:43,716 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:34:43,730 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d1f13c3aafdab4bea4e1 HTTP/1.1" 200 None 2026-03-30 01:34:43,732 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:34:43,752 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:34:43,753 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:34:43,776 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9d1f13c3aafdab4bea4e1 HTTP/1.1" 200 None 2026-03-30 01:34:43,856 - lava_callback - INFO - Completed processing callback for node 69c9d1f13c3aafdab4bea4e1 INFO: 172.20.0.1:60670 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35730 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54536 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:35:07,688 - lava_callback - INFO - Received callback for node 69c9ce933c3aafdab4bea272 INFO: 172.20.0.1:54548 - "POST /node/69c9ce933c3aafdab4bea272 HTTP/1.0" 202 Accepted 2026-03-30 01:35:07,697 - lava_callback - INFO - Starting async job submit for node 69c9ce933c3aafdab4bea272 2026-03-30 01:35:07,740 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:35:07,754 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce933c3aafdab4bea272 HTTP/1.1" 200 None 2026-03-30 01:35:07,755 - lava_callback - DEBUG - Processing node 69c9ce933c3aafdab4bea272: baseline-nfs-arm64 2026-03-30 01:35:08,857 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:35:09,026 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:35:09,027 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce933c3aafdab4bea272/log.txt.gz 2026-03-30 01:35:09,696 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:35:10,640 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:35:10,642 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce933c3aafdab4bea272/lava-logs.yaml 2026-03-30 01:35:10,664 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:35:10,840 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:35:10,841 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce933c3aafdab4bea272/lava_callback.json.gz DEBUG: Setup failed for 69c9ce933c3aafdab4bea272. Transitting job node result: incomplete -> fail 2026-03-30 01:35:10,930 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:35:10,944 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce933c3aafdab4bea272 HTTP/1.1" 200 None 2026-03-30 01:35:10,946 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:35:10,959 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:35:10,960 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:35:10,987 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce933c3aafdab4bea272 HTTP/1.1" 200 None 2026-03-30 01:35:11,097 - lava_callback - INFO - Completed processing callback for node 69c9ce933c3aafdab4bea272 2026-03-30 01:35:12,095 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:35:12,121 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:50550 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50554 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50564 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53226 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:35:26,877 - lava_callback - INFO - Received callback for node 69c7305029015d0f959925b1 INFO: 172.20.0.1:53230 - "POST /node/69c7305029015d0f959925b1 HTTP/1.0" 202 Accepted 2026-03-30 01:35:26,883 - lava_callback - INFO - Starting async job submit for node 69c7305029015d0f959925b1 2026-03-30 01:35:26,884 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:35:26,898 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7305029015d0f959925b1 HTTP/1.1" 200 None 2026-03-30 01:35:26,899 - lava_callback - DEBUG - Processing node 69c7305029015d0f959925b1: baseline-arm64-clabbe 2026-03-30 01:35:26,899 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7305029015d0f959925b1 2026-03-30 01:35:26,899 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:35:26,913 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7305029015d0f959925b1 HTTP/1.1" 200 None 2026-03-30 01:35:26,914 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:35:26,929 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b4a590270ad7b6d699d HTTP/1.1" 200 None 2026-03-30 01:35:26,930 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:35:26,950 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7305029015d0f959925b1 HTTP/1.1" 200 None 2026-03-30 01:35:26,954 - lava_callback - ERROR - Error processing callback for node 69c7305029015d0f959925b1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:39310 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37012 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56206 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39214 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46912 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46920 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46932 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51478 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:36:34,673 - lava_callback - INFO - Received callback for node 69c9cf103c3aafdab4bea3b5 INFO: 172.20.0.1:56448 - "POST /node/69c9cf103c3aafdab4bea3b5 HTTP/1.0" 202 Accepted 2026-03-30 01:36:34,721 - lava_callback - INFO - Starting async job submit for node 69c9cf103c3aafdab4bea3b5 2026-03-30 01:36:34,747 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:36:34,760 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cf103c3aafdab4bea3b5 HTTP/1.1" 200 None 2026-03-30 01:36:34,760 - lava_callback - DEBUG - Processing node 69c9cf103c3aafdab4bea3b5: baseline-arm-pengutronix 2026-03-30 01:36:34,877 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:36:35,077 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:36:35,079 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9cf103c3aafdab4bea3b5/log.txt.gz 2026-03-30 01:36:35,139 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:36:35,325 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:36:35,327 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9cf103c3aafdab4bea3b5/lava-logs.yaml 2026-03-30 01:36:35,330 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:36:35,659 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:36:35,661 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9cf103c3aafdab4bea3b5/lava_callback.json.gz DEBUG: Setup failed for 69c9cf103c3aafdab4bea3b5. Transitting job node result: incomplete -> fail 2026-03-30 01:36:35,711 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:36:35,725 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9cf103c3aafdab4bea3b5 HTTP/1.1" 200 None 2026-03-30 01:36:35,727 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:36:35,746 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 01:36:35,748 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:36:35,772 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9cf103c3aafdab4bea3b5 HTTP/1.1" 200 None 2026-03-30 01:36:35,828 - lava_callback - INFO - Completed processing callback for node 69c9cf103c3aafdab4bea3b5 INFO: 172.20.0.1:56450 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:36:42,124 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:36:42,140 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:54736 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:36:46,906 - lava_callback - INFO - Received callback for node 69c730a329015d0f959925c4 INFO: 172.20.0.1:54744 - "POST /node/69c730a329015d0f959925c4 HTTP/1.0" 202 Accepted 2026-03-30 01:36:46,912 - lava_callback - INFO - Starting async job submit for node 69c730a329015d0f959925c4 2026-03-30 01:36:46,913 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:36:46,927 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c730a329015d0f959925c4 HTTP/1.1" 200 None 2026-03-30 01:36:46,928 - lava_callback - DEBUG - Processing node 69c730a329015d0f959925c4: baseline-arm64-clabbe 2026-03-30 01:36:46,928 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c730a329015d0f959925c4 2026-03-30 01:36:46,929 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:36:46,942 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c730a329015d0f959925c4 HTTP/1.1" 200 None 2026-03-30 01:36:46,943 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:36:46,966 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b4a590270ad7b6d699d HTTP/1.1" 200 None 2026-03-30 01:36:46,968 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:36:46,995 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c730a329015d0f959925c4 HTTP/1.1" 200 None 2026-03-30 01:36:46,996 - lava_callback - ERROR - Error processing callback for node 69c730a329015d0f959925c4: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35084 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34226 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43450 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43454 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43456 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40840 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42136 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43876 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45918 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48168 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:38:06,948 - lava_callback - INFO - Received callback for node 69c730f629015d0f959925df INFO: 172.20.0.1:48174 - "POST /node/69c730f629015d0f959925df HTTP/1.0" 202 Accepted 2026-03-30 01:38:06,954 - lava_callback - INFO - Starting async job submit for node 69c730f629015d0f959925df 2026-03-30 01:38:06,955 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:38:06,968 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c730f629015d0f959925df HTTP/1.1" 200 None 2026-03-30 01:38:06,969 - lava_callback - DEBUG - Processing node 69c730f629015d0f959925df: baseline-arm64-clabbe 2026-03-30 01:38:06,969 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c730f629015d0f959925df 2026-03-30 01:38:06,969 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:38:06,982 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c730f629015d0f959925df HTTP/1.1" 200 None 2026-03-30 01:38:06,983 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:38:07,015 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b4a590270ad7b6d699d HTTP/1.1" 200 None 2026-03-30 01:38:07,016 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:38:07,037 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c730f629015d0f959925df HTTP/1.1" 200 None 2026-03-30 01:38:07,047 - lava_callback - ERROR - Error processing callback for node 69c730f629015d0f959925df: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59180 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59182 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59184 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44850 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55316 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43736 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37690 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60726 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60742 - "GET /node/69b7d52900b80c8ffdcb8c58%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:53710 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53722 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53726 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37230 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:39:27,011 - lava_callback - INFO - Received callback for node 69c7314829015d0f959925ef INFO: 172.20.0.1:37244 - "POST /node/69c7314829015d0f959925ef HTTP/1.0" 202 Accepted 2026-03-30 01:39:27,016 - lava_callback - INFO - Starting async job submit for node 69c7314829015d0f959925ef 2026-03-30 01:39:27,017 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:39:27,032 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7314829015d0f959925ef HTTP/1.1" 200 None 2026-03-30 01:39:27,032 - lava_callback - DEBUG - Processing node 69c7314829015d0f959925ef: baseline-arm64-clabbe 2026-03-30 01:39:27,033 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7314829015d0f959925ef 2026-03-30 01:39:27,033 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:39:27,059 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7314829015d0f959925ef HTTP/1.1" 200 None 2026-03-30 01:39:27,060 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:39:27,074 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b4a590270ad7b6d699d HTTP/1.1" 200 None 2026-03-30 01:39:27,076 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:39:27,097 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7314829015d0f959925ef HTTP/1.1" 200 None 2026-03-30 01:39:27,099 - lava_callback - ERROR - Error processing callback for node 69c7314829015d0f959925ef: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:50012 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57436 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35436 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55964 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60686 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60700 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60706 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50474 - "GET /node/69b3ccaa1727ff69d6f221b8%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:50490 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55412 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55068 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:40:47,084 - lava_callback - INFO - Received callback for node 69c7319b29015d0f95992602 INFO: 172.20.0.1:55074 - "POST /node/69c7319b29015d0f95992602 HTTP/1.0" 202 Accepted 2026-03-30 01:40:47,090 - lava_callback - INFO - Starting async job submit for node 69c7319b29015d0f95992602 2026-03-30 01:40:47,090 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:40:47,114 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7319b29015d0f95992602 HTTP/1.1" 200 None 2026-03-30 01:40:47,115 - lava_callback - DEBUG - Processing node 69c7319b29015d0f95992602: baseline-arm64-clabbe 2026-03-30 01:40:47,115 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7319b29015d0f95992602 2026-03-30 01:40:47,116 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:40:47,128 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7319b29015d0f95992602 HTTP/1.1" 200 None 2026-03-30 01:40:47,130 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:40:47,143 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b4a590270ad7b6d699d HTTP/1.1" 200 None 2026-03-30 01:40:47,144 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:40:47,169 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7319b29015d0f95992602 HTTP/1.1" 200 None 2026-03-30 01:40:47,174 - lava_callback - ERROR - Error processing callback for node 69c7319b29015d0f95992602: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35428 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45448 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41886 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41888 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41890 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47002 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38178 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42312 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55208 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44552 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:42:07,155 - lava_callback - INFO - Received callback for node 69c731ec29015d0f9599260f INFO: 172.20.0.1:44568 - "POST /node/69c731ec29015d0f9599260f HTTP/1.0" 202 Accepted 2026-03-30 01:42:07,160 - lava_callback - INFO - Starting async job submit for node 69c731ec29015d0f9599260f 2026-03-30 01:42:07,161 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:42:07,180 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c731ec29015d0f9599260f HTTP/1.1" 200 None 2026-03-30 01:42:07,181 - lava_callback - DEBUG - Processing node 69c731ec29015d0f9599260f: baseline-arm64-clabbe 2026-03-30 01:42:07,181 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c731ec29015d0f9599260f 2026-03-30 01:42:07,181 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:42:07,195 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c731ec29015d0f9599260f HTTP/1.1" 200 None 2026-03-30 01:42:07,196 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:42:07,209 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b4a590270ad7b6d699d HTTP/1.1" 200 None 2026-03-30 01:42:07,211 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:42:07,231 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c731ec29015d0f9599260f HTTP/1.1" 200 None 2026-03-30 01:42:07,232 - lava_callback - ERROR - Error processing callback for node 69c731ec29015d0f9599260f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57312 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57316 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34428 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38536 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32794 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41336 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60700 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49690 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49700 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49712 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56492 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51674 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40788 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:43:47,187 - lava_callback - INFO - Received callback for node 69c7323f29015d0f9599261b INFO: 172.20.0.1:40800 - "POST /node/69c7323f29015d0f9599261b HTTP/1.0" 202 Accepted 2026-03-30 01:43:47,193 - lava_callback - INFO - Starting async job submit for node 69c7323f29015d0f9599261b 2026-03-30 01:43:47,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:43:47,213 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7323f29015d0f9599261b HTTP/1.1" 200 None 2026-03-30 01:43:47,214 - lava_callback - DEBUG - Processing node 69c7323f29015d0f9599261b: baseline-arm64-clabbe 2026-03-30 01:43:47,214 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7323f29015d0f9599261b 2026-03-30 01:43:47,215 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:43:47,228 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7323f29015d0f9599261b HTTP/1.1" 200 None 2026-03-30 01:43:47,230 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:43:47,242 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 01:43:47,244 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:43:47,265 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7323f29015d0f9599261b HTTP/1.1" 200 None 2026-03-30 01:43:47,273 - lava_callback - ERROR - Error processing callback for node 69c7323f29015d0f9599261b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43110 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37276 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56318 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56326 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56340 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54486 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44922 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36638 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49376 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51612 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:45:07,269 - lava_callback - INFO - Received callback for node 69c7329129015d0f9599262b INFO: 172.20.0.1:51614 - "POST /node/69c7329129015d0f9599262b HTTP/1.0" 202 Accepted 2026-03-30 01:45:07,275 - lava_callback - INFO - Starting async job submit for node 69c7329129015d0f9599262b 2026-03-30 01:45:07,276 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:45:07,289 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7329129015d0f9599262b HTTP/1.1" 200 None 2026-03-30 01:45:07,290 - lava_callback - DEBUG - Processing node 69c7329129015d0f9599262b: baseline-arm64-clabbe 2026-03-30 01:45:07,290 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7329129015d0f9599262b 2026-03-30 01:45:07,291 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:45:07,304 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7329129015d0f9599262b HTTP/1.1" 200 None 2026-03-30 01:45:07,306 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:45:07,319 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 01:45:07,320 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:45:07,341 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7329129015d0f9599262b HTTP/1.1" 200 None 2026-03-30 01:45:07,342 - lava_callback - ERROR - Error processing callback for node 69c7329129015d0f9599262b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48656 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48672 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48674 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36318 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50088 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39878 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59042 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41304 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37568 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37572 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37588 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48468 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:46:27,285 - lava_callback - INFO - Received callback for node 69c732e129015d0f95992637 INFO: 172.20.0.1:48470 - "POST /node/69c732e129015d0f95992637 HTTP/1.0" 202 Accepted 2026-03-30 01:46:27,291 - lava_callback - INFO - Starting async job submit for node 69c732e129015d0f95992637 2026-03-30 01:46:27,292 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:46:27,335 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c732e129015d0f95992637 HTTP/1.1" 200 None 2026-03-30 01:46:27,336 - lava_callback - DEBUG - Processing node 69c732e129015d0f95992637: baseline-arm64-clabbe 2026-03-30 01:46:27,337 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c732e129015d0f95992637 2026-03-30 01:46:27,337 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:46:27,350 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c732e129015d0f95992637 HTTP/1.1" 200 None 2026-03-30 01:46:27,351 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:46:27,372 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 01:46:27,374 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:46:27,394 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c732e129015d0f95992637 HTTP/1.1" 200 None 2026-03-30 01:46:27,395 - lava_callback - ERROR - Error processing callback for node 69c732e129015d0f95992637: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35360 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59644 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56136 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41460 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41708 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41724 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41736 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58400 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33386 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51894 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:47:47,371 - lava_callback - INFO - Received callback for node 69c7333429015d0f95992645 INFO: 172.20.0.1:51896 - "POST /node/69c7333429015d0f95992645 HTTP/1.0" 202 Accepted 2026-03-30 01:47:47,376 - lava_callback - INFO - Starting async job submit for node 69c7333429015d0f95992645 2026-03-30 01:47:47,377 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:47:47,391 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7333429015d0f95992645 HTTP/1.1" 200 None 2026-03-30 01:47:47,392 - lava_callback - DEBUG - Processing node 69c7333429015d0f95992645: baseline-arm64-clabbe 2026-03-30 01:47:47,392 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7333429015d0f95992645 2026-03-30 01:47:47,393 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:47:47,406 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7333429015d0f95992645 HTTP/1.1" 200 None 2026-03-30 01:47:47,407 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:47:47,421 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 01:47:47,422 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:47:47,442 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7333429015d0f95992645 HTTP/1.1" 200 None 2026-03-30 01:47:47,446 - lava_callback - ERROR - Error processing callback for node 69c7333429015d0f95992645: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55422 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35080 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39444 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39460 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39470 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45300 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44056 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:48:39,712 - lava_callback - INFO - Received callback for node 69c9d63c3c3aafdab4bea58b INFO: 172.20.0.1:46958 - "POST /node/69c9d63c3c3aafdab4bea58b HTTP/1.0" 202 Accepted 2026-03-30 01:48:39,719 - lava_callback - INFO - Starting async job submit for node 69c9d63c3c3aafdab4bea58b 2026-03-30 01:48:39,748 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:48:39,762 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d63c3c3aafdab4bea58b HTTP/1.1" 200 None 2026-03-30 01:48:39,763 - lava_callback - DEBUG - Processing node 69c9d63c3c3aafdab4bea58b: baseline-arm64 2026-03-30 01:48:39,894 - lava_callback - WARNING - Failed to upload log 2026-03-30 01:48:39,981 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:48:40,199 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:48:40,201 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d63c3c3aafdab4bea58b/lava-logs.yaml 2026-03-30 01:48:40,205 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:48:40,348 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:48:40,350 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d63c3c3aafdab4bea58b/lava_callback.json.gz 2026-03-30 01:48:40,381 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:48:40,400 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d63c3c3aafdab4bea58b HTTP/1.1" 200 None 2026-03-30 01:48:40,402 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:48:40,418 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:48:40,419 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:48:40,446 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9d63c3c3aafdab4bea58b HTTP/1.1" 200 None 2026-03-30 01:48:40,516 - lava_callback - INFO - Completed processing callback for node 69c9d63c3c3aafdab4bea58b 2026-03-30 01:48:42,144 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:48:42,160 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:46966 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44232 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57040 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48646 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48658 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48668 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51016 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53382 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57306 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35658 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:50:03,308 - lava_callback - INFO - Received callback for node 69c9d68c3c3aafdab4bea59a INFO: 172.20.0.1:53698 - "POST /node/69c9d68c3c3aafdab4bea59a HTTP/1.0" 202 Accepted 2026-03-30 01:50:03,315 - lava_callback - INFO - Starting async job submit for node 69c9d68c3c3aafdab4bea59a 2026-03-30 01:50:03,345 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:50:03,373 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d68c3c3aafdab4bea59a HTTP/1.1" 200 None 2026-03-30 01:50:03,373 - lava_callback - DEBUG - Processing node 69c9d68c3c3aafdab4bea59a: baseline-arm64 2026-03-30 01:50:03,582 - lava_callback - WARNING - Failed to upload log 2026-03-30 01:50:03,682 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:50:03,918 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:50:03,920 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d68c3c3aafdab4bea59a/lava-logs.yaml 2026-03-30 01:50:03,925 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:50:04,059 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:50:04,061 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d68c3c3aafdab4bea59a/lava_callback.json.gz 2026-03-30 01:50:04,092 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:50:04,106 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d68c3c3aafdab4bea59a HTTP/1.1" 200 None 2026-03-30 01:50:04,108 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:50:04,135 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:50:04,136 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:50:04,167 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9d68c3c3aafdab4bea59a HTTP/1.1" 200 None 2026-03-30 01:50:04,236 - lava_callback - INFO - Completed processing callback for node 69c9d68c3c3aafdab4bea59a INFO: 172.20.0.1:53710 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:50:12,163 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:50:12,179 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:37608 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37610 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37624 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37490 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46724 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42526 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35104 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38188 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51118 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51124 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51132 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39468 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33150 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50902 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56882 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57542 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:52:13,468 - lava_callback - INFO - Received callback for node 69c9d6de3c3aafdab4bea5a9 INFO: 172.20.0.1:34018 - "POST /node/69c9d6de3c3aafdab4bea5a9 HTTP/1.0" 202 Accepted 2026-03-30 01:52:13,474 - lava_callback - INFO - Starting async job submit for node 69c9d6de3c3aafdab4bea5a9 2026-03-30 01:52:13,496 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:52:13,514 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d6de3c3aafdab4bea5a9 HTTP/1.1" 200 None 2026-03-30 01:52:13,514 - lava_callback - DEBUG - Processing node 69c9d6de3c3aafdab4bea5a9: baseline-arm64 2026-03-30 01:52:13,936 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:52:14,122 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:52:14,123 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d6de3c3aafdab4bea5a9/log.txt.gz 2026-03-30 01:52:14,407 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:52:14,645 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:52:14,647 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d6de3c3aafdab4bea5a9/lava-logs.yaml 2026-03-30 01:52:14,656 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:52:14,824 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:52:14,826 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d6de3c3aafdab4bea5a9/lava_callback.json.gz 2026-03-30 01:52:14,870 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:52:14,895 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d6de3c3aafdab4bea5a9 HTTP/1.1" 200 None 2026-03-30 01:52:14,897 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:52:14,911 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:52:14,912 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:52:14,943 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9d6de3c3aafdab4bea5a9 HTTP/1.1" 200 None 2026-03-30 01:52:15,003 - lava_callback - INFO - Completed processing callback for node 69c9d6de3c3aafdab4bea5a9 INFO: 172.20.0.1:34030 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34032 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34034 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45170 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40714 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:52:42,181 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:52:42,197 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:52464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46776 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35986 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:53:07,788 - lava_callback - INFO - Received callback for node 69c7347329015d0f9599268e INFO: 172.20.0.1:36002 - "POST /node/69c7347329015d0f9599268e HTTP/1.0" 202 Accepted 2026-03-30 01:53:07,794 - lava_callback - INFO - Starting async job submit for node 69c7347329015d0f9599268e 2026-03-30 01:53:07,795 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:53:07,811 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7347329015d0f9599268e HTTP/1.1" 200 None 2026-03-30 01:53:07,812 - lava_callback - DEBUG - Processing node 69c7347329015d0f9599268e: baseline-arm64-clabbe 2026-03-30 01:53:07,812 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7347329015d0f9599268e 2026-03-30 01:53:07,813 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:53:07,827 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7347329015d0f9599268e HTTP/1.1" 200 None 2026-03-30 01:53:07,829 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:53:07,844 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 01:53:07,846 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:53:07,882 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7347329015d0f9599268e HTTP/1.1" 200 None 2026-03-30 01:53:07,884 - lava_callback - ERROR - Error processing callback for node 69c7347329015d0f9599268e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:42186 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42188 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42192 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52024 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58146 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37506 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35418 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:54:06,349 - lava_callback - INFO - Received callback for node 69c9d7903c3aafdab4bea5ca INFO: 172.20.0.1:54260 - "POST /node/69c9d7903c3aafdab4bea5ca HTTP/1.0" 202 Accepted 2026-03-30 01:54:06,355 - lava_callback - INFO - Starting async job submit for node 69c9d7903c3aafdab4bea5ca 2026-03-30 01:54:06,374 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:54:06,395 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d7903c3aafdab4bea5ca HTTP/1.1" 200 None 2026-03-30 01:54:06,396 - lava_callback - DEBUG - Processing node 69c9d7903c3aafdab4bea5ca: baseline-arm64-pengutronix INFO: 172.20.0.1:54272 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:54:06,580 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:54:06,720 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:54:06,721 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c9d7903c3aafdab4bea5ca/log.txt.gz 2026-03-30 01:54:06,824 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:54:07,038 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:54:07,040 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c9d7903c3aafdab4bea5ca/lava-logs.yaml 2026-03-30 01:54:07,045 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:54:07,366 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:54:07,368 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69c9d7903c3aafdab4bea5ca/lava_callback.json.gz 2026-03-30 01:54:07,388 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:54:07,402 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d7903c3aafdab4bea5ca HTTP/1.1" 200 None 2026-03-30 01:54:07,403 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:54:07,417 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:54:07,419 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:54:07,449 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9d7903c3aafdab4bea5ca HTTP/1.1" 200 None 2026-03-30 01:54:07,460 - lava_callback - INFO - Completed processing callback for node 69c9d7903c3aafdab4bea5ca 2026-03-30 01:54:12,200 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:54:12,217 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:53940 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53942 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53956 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37588 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:54:35,976 - lava_callback - INFO - Received callback for node 69c9d7cf3c3aafdab4bea5d9 INFO: 172.20.0.1:34052 - "POST /node/69c9d7cf3c3aafdab4bea5d9 HTTP/1.0" 202 Accepted 2026-03-30 01:54:35,982 - lava_callback - INFO - Starting async job submit for node 69c9d7cf3c3aafdab4bea5d9 2026-03-30 01:54:35,984 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:54:35,998 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d7cf3c3aafdab4bea5d9 HTTP/1.1" 200 None 2026-03-30 01:54:35,999 - lava_callback - DEBUG - Processing node 69c9d7cf3c3aafdab4bea5d9: baseline-nfs-arm64 2026-03-30 01:54:36,001 - lava_callback - WARNING - Failed to upload log 2026-03-30 01:54:36,003 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:54:36,291 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:54:36,293 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9d7cf3c3aafdab4bea5d9/lava-logs.yaml 2026-03-30 01:54:36,294 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 01:54:36,434 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 01:54:36,436 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9d7cf3c3aafdab4bea5d9/lava_callback.json.gz 2026-03-30 01:54:36,438 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:54:36,452 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d7cf3c3aafdab4bea5d9 HTTP/1.1" 200 None 2026-03-30 01:54:36,454 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:54:36,467 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 01:54:36,469 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:54:36,497 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9d7cf3c3aafdab4bea5d9 HTTP/1.1" 200 None 2026-03-30 01:54:36,501 - lava_callback - INFO - Completed processing callback for node 69c9d7cf3c3aafdab4bea5d9 INFO: 172.20.0.1:34064 - "GET / HTTP/1.0" 200 OK 2026-03-30 01:54:42,219 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 01:54:42,236 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:51836 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37606 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40380 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48400 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48412 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48418 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53020 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53716 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49582 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37050 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51738 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45396 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45404 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45420 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60070 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37084 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47340 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58162 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60088 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45358 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45368 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45378 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51152 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54960 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43374 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34516 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33892 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33902 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33908 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60286 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59520 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38502 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33242 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49350 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32964 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:32980 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32982 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38584 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59190 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36324 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52662 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51774 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43616 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43624 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43626 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54048 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52204 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40034 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48090 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57754 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:01:07,994 - lava_callback - INFO - Received callback for node 69c7365729015d0f959926f6 INFO: 172.20.0.1:57764 - "POST /node/69c7365729015d0f959926f6 HTTP/1.0" 202 Accepted 2026-03-30 02:01:07,999 - lava_callback - INFO - Starting async job submit for node 69c7365729015d0f959926f6 2026-03-30 02:01:08,000 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:01:08,018 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7365729015d0f959926f6 HTTP/1.1" 200 None 2026-03-30 02:01:08,019 - lava_callback - DEBUG - Processing node 69c7365729015d0f959926f6: baseline-arm64-clabbe 2026-03-30 02:01:08,019 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7365729015d0f959926f6 2026-03-30 02:01:08,020 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:01:08,033 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7365729015d0f959926f6 HTTP/1.1" 200 None 2026-03-30 02:01:08,034 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:01:08,047 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 02:01:08,049 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:01:08,078 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7365729015d0f959926f6 HTTP/1.1" 200 None 2026-03-30 02:01:08,080 - lava_callback - ERROR - Error processing callback for node 69c7365729015d0f959926f6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57276 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57284 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57298 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55168 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49376 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47792 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39306 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46792 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48470 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48474 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48478 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59050 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54626 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57170 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55378 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43778 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47696 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47700 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47706 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52502 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57534 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51492 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59648 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56310 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46910 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46918 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46924 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52748 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38598 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56160 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42390 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34036 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33532 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33544 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40820 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37604 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55316 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42528 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39784 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40248 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40262 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40278 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40038 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50180 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38638 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57300 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52918 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40504 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40512 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40518 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36980 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:07:26,869 - lava_callback - INFO - Received callback for node 69c737d429015d0f95992755 INFO: 172.20.0.1:36988 - "POST /node/69c737d429015d0f95992755 HTTP/1.0" 202 Accepted 2026-03-30 02:07:26,874 - lava_callback - INFO - Starting async job submit for node 69c737d429015d0f95992755 2026-03-30 02:07:26,875 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:07:26,890 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c737d429015d0f95992755 HTTP/1.1" 200 None 2026-03-30 02:07:26,891 - lava_callback - DEBUG - Processing node 69c737d429015d0f95992755: baseline-arm64-baylibre 2026-03-30 02:07:26,892 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c737d429015d0f95992755 2026-03-30 02:07:26,892 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:07:26,907 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c737d429015d0f95992755 HTTP/1.1" 200 None 2026-03-30 02:07:26,909 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:07:26,928 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69bf3545f1004e82292e0dc6 HTTP/1.1" 200 None 2026-03-30 02:07:26,929 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:07:26,951 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c737d429015d0f95992755 HTTP/1.1" 200 None 2026-03-30 02:07:26,953 - lava_callback - ERROR - Error processing callback for node 69c737d429015d0f95992755: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45160 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50636 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37960 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:07:59,874 - lava_callback - INFO - Received callback for node 69c9d6d93c3aafdab4bea5a6 INFO: 172.20.0.1:51332 - "POST /node/69c9d6d93c3aafdab4bea5a6 HTTP/1.0" 202 Accepted 2026-03-30 02:07:59,880 - lava_callback - INFO - Starting async job submit for node 69c9d6d93c3aafdab4bea5a6 2026-03-30 02:07:59,939 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:07:59,953 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d6d93c3aafdab4bea5a6 HTTP/1.1" 200 None 2026-03-30 02:07:59,954 - lava_callback - DEBUG - Processing node 69c9d6d93c3aafdab4bea5a6: ltp-smoke 2026-03-30 02:08:00,451 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:08:00,739 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:08:00,741 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69c9d6d93c3aafdab4bea5a6/log.txt.gz 2026-03-30 02:08:01,029 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:08:01,264 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:08:01,266 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69c9d6d93c3aafdab4bea5a6/lava-logs.yaml 2026-03-30 02:08:01,278 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:08:01,464 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:08:01,465 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69c9d6d93c3aafdab4bea5a6/lava_callback.json.gz 2026-03-30 02:08:01,585 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:08:01,609 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d6d93c3aafdab4bea5a6 HTTP/1.1" 200 None 2026-03-30 02:08:01,611 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:08:01,624 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1213c3aafdab4be774c HTTP/1.1" 200 None 2026-03-30 02:08:01,625 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:08:01,650 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9d6d93c3aafdab4bea5a6 HTTP/1.1" 200 None 2026-03-30 02:08:01,778 - lava_callback - INFO - Completed processing callback for node 69c9d6d93c3aafdab4bea5a6 INFO: 172.20.0.1:51342 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:08:12,240 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:08:12,270 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:56448 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56458 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56590 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52696 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55852 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:08:46,897 - lava_callback - INFO - Received callback for node 69c7382a29015d0f9599275b INFO: 172.20.0.1:55862 - "POST /node/69c7382a29015d0f9599275b HTTP/1.0" 202 Accepted 2026-03-30 02:08:46,903 - lava_callback - INFO - Starting async job submit for node 69c7382a29015d0f9599275b 2026-03-30 02:08:46,903 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:08:46,919 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7382a29015d0f9599275b HTTP/1.1" 200 None 2026-03-30 02:08:46,920 - lava_callback - DEBUG - Processing node 69c7382a29015d0f9599275b: baseline-arm64-baylibre 2026-03-30 02:08:46,920 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7382a29015d0f9599275b 2026-03-30 02:08:46,920 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:08:46,934 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7382a29015d0f9599275b HTTP/1.1" 200 None 2026-03-30 02:08:46,936 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:08:46,950 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69bf3545f1004e82292e0dc6 HTTP/1.1" 200 None 2026-03-30 02:08:46,952 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:08:46,978 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7382a29015d0f9599275b HTTP/1.1" 200 None 2026-03-30 02:08:46,979 - lava_callback - ERROR - Error processing callback for node 69c7382a29015d0f9599275b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45488 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:09:03,235 - lava_callback - INFO - Received callback for node 69c9d98e3c3aafdab4bea618 INFO: 172.20.0.1:44638 - "POST /node/69c9d98e3c3aafdab4bea618 HTTP/1.0" 202 Accepted 2026-03-30 02:09:03,242 - lava_callback - INFO - Starting async job submit for node 69c9d98e3c3aafdab4bea618 2026-03-30 02:09:03,277 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:09:03,308 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d98e3c3aafdab4bea618 HTTP/1.1" 200 None 2026-03-30 02:09:03,309 - lava_callback - DEBUG - Processing node 69c9d98e3c3aafdab4bea618: baseline-arm64 2026-03-30 02:09:03,712 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:09:04,000 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:09:04,002 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d98e3c3aafdab4bea618/log.txt.gz 2026-03-30 02:09:04,220 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:09:04,429 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:09:04,432 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d98e3c3aafdab4bea618/lava-logs.yaml 2026-03-30 02:09:04,441 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:09:04,621 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:09:04,623 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9d98e3c3aafdab4bea618/lava_callback.json.gz DEBUG: Setup failed for 69c9d98e3c3aafdab4bea618. Transitting job node result: incomplete -> fail 2026-03-30 02:09:04,699 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:09:04,713 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d98e3c3aafdab4bea618 HTTP/1.1" 200 None 2026-03-30 02:09:04,715 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:09:04,728 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 02:09:04,730 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:09:04,753 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9d98e3c3aafdab4bea618 HTTP/1.1" 200 None 2026-03-30 02:09:04,846 - lava_callback - INFO - Completed processing callback for node 69c9d98e3c3aafdab4bea618 INFO: 172.20.0.1:44644 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:09:12,272 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:09:12,288 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:46040 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46048 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46060 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48280 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48132 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45468 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40904 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59538 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44076 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44092 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44098 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48066 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54232 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46108 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:10:47,038 - lava_callback - INFO - Received callback for node 69c738a429015d0f95992778 INFO: 172.20.0.1:46110 - "POST /node/69c738a429015d0f95992778 HTTP/1.0" 202 Accepted 2026-03-30 02:10:47,043 - lava_callback - INFO - Starting async job submit for node 69c738a429015d0f95992778 2026-03-30 02:10:47,044 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:10:47,059 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c738a429015d0f95992778 HTTP/1.1" 200 None 2026-03-30 02:10:47,060 - lava_callback - DEBUG - Processing node 69c738a429015d0f95992778: baseline-arm64-baylibre 2026-03-30 02:10:47,060 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c738a429015d0f95992778 2026-03-30 02:10:47,061 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:10:47,082 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c738a429015d0f95992778 HTTP/1.1" 200 None 2026-03-30 02:10:47,083 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:10:47,097 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69bf3545f1004e82292e0dc6 HTTP/1.1" 200 None 2026-03-30 02:10:47,098 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:10:47,119 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c738a429015d0f95992778 HTTP/1.1" 200 None 2026-03-30 02:10:47,128 - lava_callback - ERROR - Error processing callback for node 69c738a429015d0f95992778: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:42144 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55012 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33154 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33158 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33162 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40084 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47060 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40512 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41424 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57962 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33628 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33634 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33642 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54432 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:12:28,484 - lava_callback - INFO - Received callback for node 69c738fe29015d0f95992782 INFO: 172.20.0.1:54448 - "POST /node/69c738fe29015d0f95992782 HTTP/1.0" 202 Accepted 2026-03-30 02:12:28,490 - lava_callback - INFO - Starting async job submit for node 69c738fe29015d0f95992782 2026-03-30 02:12:28,491 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:12:28,505 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c738fe29015d0f95992782 HTTP/1.1" 200 None 2026-03-30 02:12:28,506 - lava_callback - DEBUG - Processing node 69c738fe29015d0f95992782: baseline-arm64-clabbe 2026-03-30 02:12:28,506 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c738fe29015d0f95992782 2026-03-30 02:12:28,506 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:12:28,533 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c738fe29015d0f95992782 HTTP/1.1" 200 None 2026-03-30 02:12:28,534 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:12:28,547 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69bf3545f1004e82292e0dc6 HTTP/1.1" 200 None 2026-03-30 02:12:28,548 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:12:28,575 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c738fe29015d0f95992782 HTTP/1.1" 200 None 2026-03-30 02:12:28,577 - lava_callback - ERROR - Error processing callback for node 69c738fe29015d0f95992782: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:39158 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54024 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52758 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47710 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39692 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39706 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39712 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37804 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51614 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37416 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:13:48,513 - lava_callback - INFO - Received callback for node 69c7395229015d0f95992793 INFO: 172.20.0.1:37422 - "POST /node/69c7395229015d0f95992793 HTTP/1.0" 202 Accepted 2026-03-30 02:13:48,519 - lava_callback - INFO - Starting async job submit for node 69c7395229015d0f95992793 2026-03-30 02:13:48,520 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:13:48,536 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7395229015d0f95992793 HTTP/1.1" 200 None 2026-03-30 02:13:48,537 - lava_callback - DEBUG - Processing node 69c7395229015d0f95992793: baseline-arm64-clabbe 2026-03-30 02:13:48,537 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7395229015d0f95992793 2026-03-30 02:13:48,538 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:13:48,551 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7395229015d0f95992793 HTTP/1.1" 200 None 2026-03-30 02:13:48,553 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:13:48,566 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69bf3545f1004e82292e0dc6 HTTP/1.1" 200 None 2026-03-30 02:13:48,568 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:13:48,588 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7395229015d0f95992793 HTTP/1.1" 200 None 2026-03-30 02:13:48,590 - lava_callback - ERROR - Error processing callback for node 69c7395229015d0f95992793: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46962 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42100 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35300 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35302 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35310 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48152 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53108 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46190 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34430 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52642 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:15:08,574 - lava_callback - INFO - Received callback for node 69c739a729015d0f959927a0 INFO: 172.20.0.1:52652 - "POST /node/69c739a729015d0f959927a0 HTTP/1.0" 202 Accepted 2026-03-30 02:15:08,579 - lava_callback - INFO - Starting async job submit for node 69c739a729015d0f959927a0 2026-03-30 02:15:08,580 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:15:08,604 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c739a729015d0f959927a0 HTTP/1.1" 200 None 2026-03-30 02:15:08,605 - lava_callback - DEBUG - Processing node 69c739a729015d0f959927a0: baseline-arm64-clabbe 2026-03-30 02:15:08,605 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c739a729015d0f959927a0 2026-03-30 02:15:08,606 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:15:08,619 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c739a729015d0f959927a0 HTTP/1.1" 200 None 2026-03-30 02:15:08,620 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:15:08,633 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69bf3545f1004e82292e0dc6 HTTP/1.1" 200 None 2026-03-30 02:15:08,634 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:15:08,654 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c739a729015d0f959927a0 HTTP/1.1" 200 None 2026-03-30 02:15:08,655 - lava_callback - ERROR - Error processing callback for node 69c739a729015d0f959927a0: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48392 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48404 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48414 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43514 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48506 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47560 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40514 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39812 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48360 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48362 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48366 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43192 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34576 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33776 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:16:48,647 - lava_callback - INFO - Received callback for node 69c739fd29015d0f959927ab INFO: 172.20.0.1:33788 - "POST /node/69c739fd29015d0f959927ab HTTP/1.0" 202 Accepted 2026-03-30 02:16:48,652 - lava_callback - INFO - Starting async job submit for node 69c739fd29015d0f959927ab 2026-03-30 02:16:48,653 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:16:48,668 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c739fd29015d0f959927ab HTTP/1.1" 200 None 2026-03-30 02:16:48,669 - lava_callback - DEBUG - Processing node 69c739fd29015d0f959927ab: baseline-arm64-clabbe 2026-03-30 02:16:48,669 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c739fd29015d0f959927ab 2026-03-30 02:16:48,670 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:16:48,683 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c739fd29015d0f959927ab HTTP/1.1" 200 None 2026-03-30 02:16:48,684 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:16:48,698 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69bf3545f1004e82292e0dc6 HTTP/1.1" 200 None 2026-03-30 02:16:48,699 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:16:48,721 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c739fd29015d0f959927ab HTTP/1.1" 200 None 2026-03-30 02:16:48,730 - lava_callback - ERROR - Error processing callback for node 69c739fd29015d0f959927ab: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46972 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60614 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47310 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47312 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47318 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40128 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51508 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39922 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49328 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:17:58,921 - lava_callback - INFO - Received callback for node 69c9d9153c3aafdab4bea602 INFO: 172.20.0.1:49334 - "POST /node/69c9d9153c3aafdab4bea602 HTTP/1.0" 202 Accepted 2026-03-30 02:17:58,928 - lava_callback - INFO - Starting async job submit for node 69c9d9153c3aafdab4bea602 2026-03-30 02:17:58,981 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:17:59,005 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d9153c3aafdab4bea602 HTTP/1.1" 200 None 2026-03-30 02:17:59,006 - lava_callback - DEBUG - Processing node 69c9d9153c3aafdab4bea602: baseline-arm 2026-03-30 02:17:59,618 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:17:59,921 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:17:59,923 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69c9d9153c3aafdab4bea602/log.txt.gz 2026-03-30 02:18:00,276 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:18:00,536 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:18:00,539 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c9d9153c3aafdab4bea602/lava-logs.yaml 2026-03-30 02:18:00,553 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:18:00,735 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:18:00,737 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c9d9153c3aafdab4bea602/lava_callback.json.gz DEBUG: Setup failed for 69c9d9153c3aafdab4bea602. Transitting job node result: incomplete -> fail 2026-03-30 02:18:00,846 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:18:00,860 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d9153c3aafdab4bea602 HTTP/1.1" 200 None 2026-03-30 02:18:00,862 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:18:00,875 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 02:18:00,877 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:18:00,903 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9d9153c3aafdab4bea602 HTTP/1.1" 200 None 2026-03-30 02:18:01,018 - lava_callback - INFO - Completed processing callback for node 69c9d9153c3aafdab4bea602 INFO: 172.20.0.1:36186 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:18:12,292 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:18:12,343 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:39032 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39044 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39054 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52798 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57960 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40520 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46064 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46704 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34938 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34954 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34956 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38090 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:19:28,786 - lava_callback - INFO - Received callback for node 69c73aa229015d0f959927c2 INFO: 172.20.0.1:38092 - "POST /node/69c73aa229015d0f959927c2 HTTP/1.0" 202 Accepted 2026-03-30 02:19:28,791 - lava_callback - INFO - Starting async job submit for node 69c73aa229015d0f959927c2 2026-03-30 02:19:28,792 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:19:28,807 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c73aa229015d0f959927c2 HTTP/1.1" 200 None 2026-03-30 02:19:28,808 - lava_callback - DEBUG - Processing node 69c73aa229015d0f959927c2: baseline-arm64-clabbe 2026-03-30 02:19:28,808 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c73aa229015d0f959927c2 2026-03-30 02:19:28,808 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:19:28,822 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c73aa229015d0f959927c2 HTTP/1.1" 200 None 2026-03-30 02:19:28,824 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:19:28,838 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69bf3545f1004e82292e0dc6 HTTP/1.1" 200 None 2026-03-30 02:19:28,839 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:19:28,873 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c73aa229015d0f959927c2 HTTP/1.1" 200 None 2026-03-30 02:19:28,875 - lava_callback - ERROR - Error processing callback for node 69c73aa229015d0f959927c2: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:56964 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53076 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46632 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:20:02,596 - lava_callback - INFO - Received callback for node 69c9dd943c3aafdab4bea6a8 INFO: 172.20.0.1:43284 - "POST /node/69c9dd943c3aafdab4bea6a8 HTTP/1.0" 202 Accepted 2026-03-30 02:20:02,603 - lava_callback - INFO - Starting async job submit for node 69c9dd943c3aafdab4bea6a8 2026-03-30 02:20:02,632 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:20:02,658 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9dd943c3aafdab4bea6a8 HTTP/1.1" 200 None 2026-03-30 02:20:02,659 - lava_callback - DEBUG - Processing node 69c9dd943c3aafdab4bea6a8: baseline-arm64 2026-03-30 02:20:02,795 - lava_callback - WARNING - Failed to upload log 2026-03-30 02:20:02,864 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:20:03,083 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:20:03,085 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9dd943c3aafdab4bea6a8/lava-logs.yaml 2026-03-30 02:20:03,089 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:20:03,246 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:20:03,248 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9dd943c3aafdab4bea6a8/lava_callback.json.gz 2026-03-30 02:20:03,278 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:20:03,293 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9dd943c3aafdab4bea6a8 HTTP/1.1" 200 None 2026-03-30 02:20:03,295 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:20:03,308 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 02:20:03,310 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:20:03,331 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9dd943c3aafdab4bea6a8 HTTP/1.1" 200 None 2026-03-30 02:20:03,411 - lava_callback - INFO - Completed processing callback for node 69c9dd943c3aafdab4bea6a8 INFO: 172.20.0.1:43292 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:20:12,345 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:20:12,362 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:59086 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59098 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59100 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48698 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56922 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53302 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39596 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33966 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39386 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39402 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39416 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:21:21,158 - lava_callback - INFO - Received callback for node 69c9dde43c3aafdab4bea6b1 INFO: 172.20.0.1:50246 - "POST /node/69c9dde43c3aafdab4bea6b1 HTTP/1.0" 202 Accepted 2026-03-30 02:21:21,165 - lava_callback - INFO - Starting async job submit for node 69c9dde43c3aafdab4bea6b1 2026-03-30 02:21:21,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:21:21,209 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9dde43c3aafdab4bea6b1 HTTP/1.1" 200 None 2026-03-30 02:21:21,210 - lava_callback - DEBUG - Processing node 69c9dde43c3aafdab4bea6b1: baseline-arm64 2026-03-30 02:21:21,375 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:21:21,665 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:21:21,667 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9dde43c3aafdab4bea6b1/log.txt.gz 2026-03-30 02:21:21,737 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:21:21,930 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:21:21,932 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9dde43c3aafdab4bea6b1/lava-logs.yaml 2026-03-30 02:21:21,935 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:21:22,079 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:21:22,080 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9dde43c3aafdab4bea6b1/lava_callback.json.gz 2026-03-30 02:21:22,111 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:21:22,126 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9dde43c3aafdab4bea6b1 HTTP/1.1" 200 None 2026-03-30 02:21:22,128 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:21:22,141 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 02:21:22,142 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:21:22,163 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9dde43c3aafdab4bea6b1 HTTP/1.1" 200 None 2026-03-30 02:21:22,243 - lava_callback - INFO - Completed processing callback for node 69c9dde43c3aafdab4bea6b1 INFO: 172.20.0.1:50258 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53452 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:21:42,364 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:21:42,379 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:60030 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37332 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38476 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:22:08,866 - lava_callback - INFO - Received callback for node 69c73b4c29015d0f959927da INFO: 172.20.0.1:38486 - "POST /node/69c73b4c29015d0f959927da HTTP/1.0" 202 Accepted 2026-03-30 02:22:08,872 - lava_callback - INFO - Starting async job submit for node 69c73b4c29015d0f959927da 2026-03-30 02:22:08,873 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:22:08,888 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c73b4c29015d0f959927da HTTP/1.1" 200 None 2026-03-30 02:22:08,889 - lava_callback - DEBUG - Processing node 69c73b4c29015d0f959927da: baseline-arm64-clabbe 2026-03-30 02:22:08,889 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c73b4c29015d0f959927da 2026-03-30 02:22:08,890 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:22:08,903 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c73b4c29015d0f959927da HTTP/1.1" 200 None 2026-03-30 02:22:08,905 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:22:08,926 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69bf3545f1004e82292e0dc6 HTTP/1.1" 200 None 2026-03-30 02:22:08,928 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:22:08,952 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c73b4c29015d0f959927da HTTP/1.1" 200 None 2026-03-30 02:22:08,960 - lava_callback - ERROR - Error processing callback for node 69c73b4c29015d0f959927da: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59144 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59150 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59164 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60574 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58482 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35910 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:22:51,342 - lava_callback - INFO - Received callback for node 69c9de353c3aafdab4bea6bb INFO: 172.20.0.1:59834 - "POST /node/69c9de353c3aafdab4bea6bb HTTP/1.0" 202 Accepted 2026-03-30 02:22:51,349 - lava_callback - INFO - Starting async job submit for node 69c9de353c3aafdab4bea6bb 2026-03-30 02:22:51,375 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:22:51,390 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9de353c3aafdab4bea6bb HTTP/1.1" 200 None 2026-03-30 02:22:51,391 - lava_callback - DEBUG - Processing node 69c9de353c3aafdab4bea6bb: baseline-arm64 2026-03-30 02:22:51,811 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:22:52,139 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:22:52,141 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9de353c3aafdab4bea6bb/log.txt.gz 2026-03-30 02:22:52,395 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:22:52,623 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:22:52,625 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9de353c3aafdab4bea6bb/lava-logs.yaml 2026-03-30 02:22:52,635 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:22:52,822 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:22:52,823 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9de353c3aafdab4bea6bb/lava_callback.json.gz 2026-03-30 02:22:52,869 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:22:52,883 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9de353c3aafdab4bea6bb HTTP/1.1" 200 None 2026-03-30 02:22:52,885 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:22:52,898 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 02:22:52,900 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:22:52,924 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9de353c3aafdab4bea6bb HTTP/1.1" 200 None 2026-03-30 02:22:52,973 - lava_callback - INFO - Completed processing callback for node 69c9de353c3aafdab4bea6bb INFO: 172.20.0.1:59850 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35492 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:23:12,381 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:23:12,398 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:57918 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57928 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57934 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56284 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:23:26,858 - lava_callback - INFO - Received callback for node 69c9d9e03c3aafdab4bea621 INFO: 172.20.0.1:56288 - "POST /node/69c9d9e03c3aafdab4bea621 HTTP/1.0" 202 Accepted 2026-03-30 02:23:26,866 - lava_callback - INFO - Starting async job submit for node 69c9d9e03c3aafdab4bea621 2026-03-30 02:23:26,928 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:23:26,943 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d9e03c3aafdab4bea621 HTTP/1.1" 200 None 2026-03-30 02:23:26,944 - lava_callback - DEBUG - Processing node 69c9d9e03c3aafdab4bea621: baseline-nfs-arm64 2026-03-30 02:23:28,051 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:23:28,210 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:23:28,212 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9d9e03c3aafdab4bea621/log.txt.gz 2026-03-30 02:23:28,855 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:23:29,109 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:23:29,112 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9d9e03c3aafdab4bea621/lava-logs.yaml 2026-03-30 02:23:29,134 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:23:29,477 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:23:29,479 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9d9e03c3aafdab4bea621/lava_callback.json.gz DEBUG: Setup failed for 69c9d9e03c3aafdab4bea621. Transitting job node result: incomplete -> fail 2026-03-30 02:23:29,589 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:23:29,603 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9d9e03c3aafdab4bea621 HTTP/1.1" 200 None 2026-03-30 02:23:29,604 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:23:29,618 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 02:23:29,619 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:23:29,643 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9d9e03c3aafdab4bea621 HTTP/1.1" 200 None 2026-03-30 02:23:29,730 - lava_callback - INFO - Completed processing callback for node 69c9d9e03c3aafdab4bea621 INFO: 172.20.0.1:47720 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:23:42,400 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:23:42,416 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:59076 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53926 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42894 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43542 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43544 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43554 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60076 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48198 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55634 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54166 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52226 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48136 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48144 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:25:17,745 - lava_callback - INFO - Received callback for node 69c9da843c3aafdab4bea633 INFO: 172.20.0.1:48148 - "POST /node/69c9da843c3aafdab4bea633 HTTP/1.0" 202 Accepted 2026-03-30 02:25:17,797 - lava_callback - INFO - Starting async job submit for node 69c9da843c3aafdab4bea633 2026-03-30 02:25:17,822 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:25:17,837 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9da843c3aafdab4bea633 HTTP/1.1" 200 None 2026-03-30 02:25:17,838 - lava_callback - DEBUG - Processing node 69c9da843c3aafdab4bea633: baseline-arm-pengutronix 2026-03-30 02:25:17,955 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:25:18,117 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:25:18,118 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9da843c3aafdab4bea633/log.txt.gz 2026-03-30 02:25:18,177 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:25:18,502 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:25:18,504 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9da843c3aafdab4bea633/lava-logs.yaml 2026-03-30 02:25:18,507 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:25:18,645 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:25:18,647 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9da843c3aafdab4bea633/lava_callback.json.gz DEBUG: Setup failed for 69c9da843c3aafdab4bea633. Transitting job node result: incomplete -> fail 2026-03-30 02:25:18,696 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:25:18,713 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9da843c3aafdab4bea633 HTTP/1.1" 200 None 2026-03-30 02:25:18,715 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:25:18,728 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 02:25:18,730 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:25:18,761 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9da843c3aafdab4bea633 HTTP/1.1" 200 None 2026-03-30 02:25:18,813 - lava_callback - INFO - Completed processing callback for node 69c9da843c3aafdab4bea633 INFO: 172.20.0.1:48152 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60960 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39256 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:25:42,418 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:25:42,442 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:53618 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36462 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54978 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56524 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56538 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56548 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57466 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35474 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52536 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38828 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53238 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53252 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53266 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57434 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36628 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60322 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41502 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40596 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57914 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57918 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57930 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49854 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48130 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44328 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44904 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33300 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45310 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45312 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35644 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33772 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48946 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51980 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43614 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35620 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35632 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35636 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40146 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48378 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47092 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40144 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38856 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51444 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51460 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54710 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58804 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39742 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49314 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49324 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50514 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57654 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60974 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33450 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59480 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:33:08,259 - lava_callback - INFO - Received callback for node 69c9df293c3aafdab4bea6eb INFO: 172.20.0.1:59482 - "POST /node/69c9df293c3aafdab4bea6eb HTTP/1.0" 202 Accepted 2026-03-30 02:33:08,267 - lava_callback - INFO - Starting async job submit for node 69c9df293c3aafdab4bea6eb 2026-03-30 02:33:08,307 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:33:08,360 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9df293c3aafdab4bea6eb HTTP/1.1" 200 None 2026-03-30 02:33:08,361 - lava_callback - DEBUG - Processing node 69c9df293c3aafdab4bea6eb: baseline-arm64 2026-03-30 02:33:08,728 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:33:08,904 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:33:08,906 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9df293c3aafdab4bea6eb/log.txt.gz 2026-03-30 02:33:09,120 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:33:09,374 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:33:09,376 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9df293c3aafdab4bea6eb/lava-logs.yaml 2026-03-30 02:33:09,385 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:33:09,706 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:33:09,708 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9df293c3aafdab4bea6eb/lava_callback.json.gz DEBUG: Setup failed for 69c9df293c3aafdab4bea6eb. Transitting job node result: incomplete -> fail 2026-03-30 02:33:09,802 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:33:09,818 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9df293c3aafdab4bea6eb HTTP/1.1" 200 None 2026-03-30 02:33:09,820 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:33:09,842 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 02:33:09,844 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:33:09,881 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9df293c3aafdab4bea6eb HTTP/1.1" 200 None 2026-03-30 02:33:09,951 - lava_callback - INFO - Completed processing callback for node 69c9df293c3aafdab4bea6eb 2026-03-30 02:33:12,445 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:33:12,462 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:49546 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49554 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49570 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52182 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47854 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52132 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59854 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59870 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59886 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33158 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60148 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42036 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35106 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43652 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47540 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47552 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47558 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37836 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41504 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56366 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55332 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43060 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53312 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53322 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53332 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40740 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41566 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48560 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56152 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59280 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60694 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60696 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41204 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41208 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33824 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57600 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54310 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45044 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53038 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53046 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55866 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55882 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41890 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51522 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51842 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51056 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48594 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48608 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56736 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56742 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48082 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38160 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43552 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39672 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48890 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48896 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52484 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52494 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42626 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42660 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56622 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33080 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48146 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48162 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59978 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59986 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38708 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36606 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57022 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35264 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34576 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34578 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34036 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34042 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33086 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38468 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44624 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34218 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59752 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59758 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58824 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:43:21,407 - lava_callback - INFO - Received callback for node 69c9df1f3c3aafdab4bea6e6 INFO: 172.20.0.1:58828 - "POST /node/69c9df1f3c3aafdab4bea6e6 HTTP/1.0" 202 Accepted 2026-03-30 02:43:21,414 - lava_callback - INFO - Starting async job submit for node 69c9df1f3c3aafdab4bea6e6 2026-03-30 02:43:21,491 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:43:21,507 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9df1f3c3aafdab4bea6e6 HTTP/1.1" 200 None 2026-03-30 02:43:21,508 - lava_callback - DEBUG - Processing node 69c9df1f3c3aafdab4bea6e6: ltp-smoke 2026-03-30 02:43:21,992 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:43:22,156 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:43:22,158 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69c9df1f3c3aafdab4bea6e6/log.txt.gz 2026-03-30 02:43:22,443 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:43:22,667 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:43:22,669 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69c9df1f3c3aafdab4bea6e6/lava-logs.yaml 2026-03-30 02:43:22,681 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:43:22,876 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:43:22,878 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69c9df1f3c3aafdab4bea6e6/lava_callback.json.gz 2026-03-30 02:43:23,018 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:43:23,046 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9df1f3c3aafdab4bea6e6 HTTP/1.1" 200 None 2026-03-30 02:43:23,048 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:43:23,062 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1213c3aafdab4be774c HTTP/1.1" 200 None 2026-03-30 02:43:23,063 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:43:23,088 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9df1f3c3aafdab4bea6e6 HTTP/1.1" 200 None 2026-03-30 02:43:23,204 - lava_callback - INFO - Completed processing callback for node 69c9df1f3c3aafdab4bea6e6 INFO: 172.20.0.1:58840 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56296 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:43:42,466 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:43:42,482 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:57874 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:43:49,029 - lava_callback - INFO - Received callback for node 69c7405329015d0f959928f9 INFO: 172.20.0.1:57880 - "POST /node/69c7405329015d0f959928f9 HTTP/1.0" 202 Accepted 2026-03-30 02:43:49,035 - lava_callback - INFO - Starting async job submit for node 69c7405329015d0f959928f9 2026-03-30 02:43:49,036 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:43:49,050 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7405329015d0f959928f9 HTTP/1.1" 200 None 2026-03-30 02:43:49,051 - lava_callback - DEBUG - Processing node 69c7405329015d0f959928f9: baseline-arm-baylibre 2026-03-30 02:43:49,051 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7405329015d0f959928f9 2026-03-30 02:43:49,052 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:43:49,080 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7405329015d0f959928f9 HTTP/1.1" 200 None 2026-03-30 02:43:49,082 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:43:49,102 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63ce4b98d4d3c02067dbf HTTP/1.1" 200 None 2026-03-30 02:43:49,104 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:43:49,130 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7405329015d0f959928f9 HTTP/1.1" 200 None 2026-03-30 02:43:49,140 - lava_callback - ERROR - Error processing callback for node 69c7405329015d0f959928f9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59602 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:43:58,136 - lava_callback - INFO - Received callback for node 69c9df743c3aafdab4bea6f5 INFO: 172.20.0.1:59618 - "POST /node/69c9df743c3aafdab4bea6f5 HTTP/1.0" 202 Accepted 2026-03-30 02:43:58,143 - lava_callback - INFO - Starting async job submit for node 69c9df743c3aafdab4bea6f5 2026-03-30 02:43:58,199 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:43:58,214 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9df743c3aafdab4bea6f5 HTTP/1.1" 200 None 2026-03-30 02:43:58,215 - lava_callback - DEBUG - Processing node 69c9df743c3aafdab4bea6f5: baseline-arm 2026-03-30 02:43:58,795 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:43:59,230 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:43:59,232 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69c9df743c3aafdab4bea6f5/log.txt.gz 2026-03-30 02:43:59,593 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:43:59,832 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:43:59,834 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69c9df743c3aafdab4bea6f5/lava-logs.yaml 2026-03-30 02:43:59,848 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:44:00,014 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:44:00,016 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69c9df743c3aafdab4bea6f5/lava_callback.json.gz DEBUG: Setup failed for 69c9df743c3aafdab4bea6f5. Transitting job node result: incomplete -> fail 2026-03-30 02:44:00,124 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:44:00,138 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9df743c3aafdab4bea6f5 HTTP/1.1" 200 None 2026-03-30 02:44:00,139 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:44:00,154 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 02:44:00,156 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:44:00,180 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9df743c3aafdab4bea6f5 HTTP/1.1" 200 None 2026-03-30 02:44:00,285 - lava_callback - INFO - Completed processing callback for node 69c9df743c3aafdab4bea6f5 INFO: 172.20.0.1:55502 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:44:12,484 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:44:12,499 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:48526 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48530 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52866 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52882 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59728 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54392 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51290 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42926 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49850 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49854 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58304 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58314 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:45:28,942 - lava_callback - INFO - Received callback for node 69c740be29015d0f95992922 INFO: 172.20.0.1:58330 - "POST /node/69c740be29015d0f95992922 HTTP/1.0" 202 Accepted 2026-03-30 02:45:28,948 - lava_callback - INFO - Starting async job submit for node 69c740be29015d0f95992922 2026-03-30 02:45:28,949 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:45:28,963 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c740be29015d0f95992922 HTTP/1.1" 200 None 2026-03-30 02:45:28,964 - lava_callback - DEBUG - Processing node 69c740be29015d0f95992922: baseline-arm-baylibre 2026-03-30 02:45:28,964 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c740be29015d0f95992922 2026-03-30 02:45:28,965 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:45:28,978 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c740be29015d0f95992922 HTTP/1.1" 200 None 2026-03-30 02:45:28,980 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:45:28,993 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63ce4b98d4d3c02067dbf HTTP/1.1" 200 None 2026-03-30 02:45:28,994 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:45:29,014 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c740be29015d0f95992922 HTTP/1.1" 200 None 2026-03-30 02:45:29,021 - lava_callback - ERROR - Error processing callback for node 69c740be29015d0f95992922: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44980 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47132 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57834 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54360 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44138 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44148 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57012 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57016 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35822 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40436 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46764 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51934 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59600 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59610 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44324 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38610 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53030 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:47:48,549 - lava_callback - INFO - Received callback for node 69c9df9f3c3aafdab4bea6fe INFO: 172.20.0.1:53032 - "POST /node/69c9df9f3c3aafdab4bea6fe HTTP/1.0" 202 Accepted 2026-03-30 02:47:48,557 - lava_callback - INFO - Starting async job submit for node 69c9df9f3c3aafdab4bea6fe 2026-03-30 02:47:48,600 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:47:48,644 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9df9f3c3aafdab4bea6fe HTTP/1.1" 200 None 2026-03-30 02:47:48,645 - lava_callback - DEBUG - Processing node 69c9df9f3c3aafdab4bea6fe: baseline-nfs-arm64 2026-03-30 02:47:49,757 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:47:49,962 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:47:49,964 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9df9f3c3aafdab4bea6fe/log.txt.gz 2026-03-30 02:47:50,614 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:47:50,874 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:47:50,877 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9df9f3c3aafdab4bea6fe/lava-logs.yaml 2026-03-30 02:47:50,899 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:47:51,104 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:47:51,105 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9df9f3c3aafdab4bea6fe/lava_callback.json.gz DEBUG: Setup failed for 69c9df9f3c3aafdab4bea6fe. Transitting job node result: incomplete -> fail 2026-03-30 02:47:51,197 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:47:51,215 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9df9f3c3aafdab4bea6fe HTTP/1.1" 200 None 2026-03-30 02:47:51,217 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:47:51,230 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 02:47:51,232 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:47:51,264 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9df9f3c3aafdab4bea6fe HTTP/1.1" 200 None 2026-03-30 02:47:51,368 - lava_callback - INFO - Completed processing callback for node 69c9df9f3c3aafdab4bea6fe INFO: 172.20.0.1:33048 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55578 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:48:12,502 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:48:12,518 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:41962 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41970 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34094 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34108 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49086 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:48:38,975 - lava_callback - INFO - Received callback for node 69c9dff03c3aafdab4bea709 INFO: 172.20.0.1:49094 - "POST /node/69c9dff03c3aafdab4bea709 HTTP/1.0" 202 Accepted 2026-03-30 02:48:39,021 - lava_callback - INFO - Starting async job submit for node 69c9dff03c3aafdab4bea709 2026-03-30 02:48:39,046 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:48:39,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9dff03c3aafdab4bea709 HTTP/1.1" 200 None 2026-03-30 02:48:39,073 - lava_callback - DEBUG - Processing node 69c9dff03c3aafdab4bea709: baseline-arm-pengutronix 2026-03-30 02:48:39,177 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:48:39,321 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:48:39,322 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9dff03c3aafdab4bea709/log.txt.gz 2026-03-30 02:48:39,383 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:48:39,561 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:48:39,563 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9dff03c3aafdab4bea709/lava-logs.yaml 2026-03-30 02:48:39,566 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 02:48:39,718 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 02:48:39,720 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69c9dff03c3aafdab4bea709/lava_callback.json.gz DEBUG: Setup failed for 69c9dff03c3aafdab4bea709. Transitting job node result: incomplete -> fail 2026-03-30 02:48:39,769 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:48:39,784 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9dff03c3aafdab4bea709 HTTP/1.1" 200 None 2026-03-30 02:48:39,785 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:48:39,800 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c11b3c3aafdab4be7746 HTTP/1.1" 200 None 2026-03-30 02:48:39,801 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:48:39,830 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9dff03c3aafdab4bea709 HTTP/1.1" 200 None 2026-03-30 02:48:39,893 - lava_callback - INFO - Completed processing callback for node 69c9dff03c3aafdab4bea709 2026-03-30 02:48:42,521 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:48:42,537 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:58400 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36006 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38830 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38268 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38278 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33294 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33298 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:49:30,107 - lava_callback - INFO - Received callback for node 69c741a329015d0f9599296c INFO: 172.20.0.1:58696 - "POST /node/69c741a329015d0f9599296c HTTP/1.0" 202 Accepted 2026-03-30 02:49:30,113 - lava_callback - INFO - Starting async job submit for node 69c741a329015d0f9599296c 2026-03-30 02:49:30,114 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:49:30,129 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c741a329015d0f9599296c HTTP/1.1" 200 None 2026-03-30 02:49:30,130 - lava_callback - DEBUG - Processing node 69c741a329015d0f9599296c: baseline-arm-clabbe 2026-03-30 02:49:30,130 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c741a329015d0f9599296c 2026-03-30 02:49:30,131 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:49:30,145 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c741a329015d0f9599296c HTTP/1.1" 200 None 2026-03-30 02:49:30,146 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:49:30,159 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63ce4b98d4d3c02067dbf HTTP/1.1" 200 None 2026-03-30 02:49:30,161 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:49:30,181 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c741a329015d0f9599296c HTTP/1.1" 200 None 2026-03-30 02:49:30,182 - lava_callback - ERROR - Error processing callback for node 69c741a329015d0f9599296c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:58706 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43220 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:49:50,074 - lava_callback - INFO - Received callback for node 69c741c829015d0f9599297c INFO: 172.20.0.1:52764 - "POST /node/69c741c829015d0f9599297c HTTP/1.0" 202 Accepted 2026-03-30 02:49:50,080 - lava_callback - INFO - Starting async job submit for node 69c741c829015d0f9599297c 2026-03-30 02:49:50,080 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:49:50,096 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c741c829015d0f9599297c HTTP/1.1" 200 None 2026-03-30 02:49:50,097 - lava_callback - DEBUG - Processing node 69c741c829015d0f9599297c: baseline-arm-clabbe 2026-03-30 02:49:50,097 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c741c829015d0f9599297c 2026-03-30 02:49:50,098 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:49:50,111 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c741c829015d0f9599297c HTTP/1.1" 200 None 2026-03-30 02:49:50,112 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:49:50,127 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63ce4b98d4d3c02067dbf HTTP/1.1" 200 None 2026-03-30 02:49:50,129 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:49:50,152 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c741c829015d0f9599297c HTTP/1.1" 200 None 2026-03-30 02:49:50,153 - lava_callback - ERROR - Error processing callback for node 69c741c829015d0f9599297c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:52780 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38834 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47324 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47336 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55342 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58836 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50834 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:50:50,163 - lava_callback - INFO - Received callback for node 69c741f829015d0f9599298a INFO: 172.20.0.1:55792 - "POST /node/69c741f829015d0f9599298a HTTP/1.0" 202 Accepted 2026-03-30 02:50:50,169 - lava_callback - INFO - Starting async job submit for node 69c741f829015d0f9599298a 2026-03-30 02:50:50,170 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:50:50,185 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c741f829015d0f9599298a HTTP/1.1" 200 None 2026-03-30 02:50:50,185 - lava_callback - DEBUG - Processing node 69c741f829015d0f9599298a: baseline-arm-clabbe 2026-03-30 02:50:50,186 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c741f829015d0f9599298a 2026-03-30 02:50:50,186 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:50:50,199 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c741f829015d0f9599298a HTTP/1.1" 200 None 2026-03-30 02:50:50,200 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:50:50,213 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63ce4b98d4d3c02067dbf HTTP/1.1" 200 None 2026-03-30 02:50:50,215 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:50:50,239 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c741f829015d0f9599298a HTTP/1.1" 200 None 2026-03-30 02:50:50,245 - lava_callback - ERROR - Error processing callback for node 69c741f829015d0f9599298a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55808 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51528 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54932 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54940 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49676 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49678 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36218 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:51:50,237 - lava_callback - INFO - Received callback for node 69c7423429015d0f95992999 INFO: 172.20.0.1:57838 - "POST /node/69c7423429015d0f95992999 HTTP/1.0" 202 Accepted 2026-03-30 02:51:50,242 - lava_callback - INFO - Starting async job submit for node 69c7423429015d0f95992999 2026-03-30 02:51:50,243 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:51:50,274 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7423429015d0f95992999 HTTP/1.1" 200 None 2026-03-30 02:51:50,275 - lava_callback - DEBUG - Processing node 69c7423429015d0f95992999: baseline-arm-clabbe 2026-03-30 02:51:50,275 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7423429015d0f95992999 2026-03-30 02:51:50,276 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:51:50,300 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7423429015d0f95992999 HTTP/1.1" 200 None 2026-03-30 02:51:50,302 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:51:50,315 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63ce4b98d4d3c02067dbf HTTP/1.1" 200 None 2026-03-30 02:51:50,316 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:51:50,357 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7423429015d0f95992999 HTTP/1.1" 200 None 2026-03-30 02:51:50,358 - lava_callback - ERROR - Error processing callback for node 69c7423429015d0f95992999: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57846 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48786 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52324 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52340 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38268 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38274 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40132 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58634 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56800 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33564 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48644 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48652 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41340 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41344 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48196 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44296 - "GET / HTTP/1.0" 200 OK 2026-03-30 02:53:50,313 - lava_callback - INFO - Received callback for node 69c742ad29015d0f959929ab INFO: 172.20.0.1:56564 - "POST /node/69c742ad29015d0f959929ab HTTP/1.0" 202 Accepted 2026-03-30 02:53:50,319 - lava_callback - INFO - Starting async job submit for node 69c742ad29015d0f959929ab 2026-03-30 02:53:50,320 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:53:50,336 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c742ad29015d0f959929ab HTTP/1.1" 200 None 2026-03-30 02:53:50,337 - lava_callback - DEBUG - Processing node 69c742ad29015d0f959929ab: baseline-arm-clabbe 2026-03-30 02:53:50,337 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c742ad29015d0f959929ab 2026-03-30 02:53:50,338 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:53:50,351 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c742ad29015d0f959929ab HTTP/1.1" 200 None 2026-03-30 02:53:50,352 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:53:50,365 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63ce4b98d4d3c02067dbf HTTP/1.1" 200 None 2026-03-30 02:53:50,366 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 02:53:50,388 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c742ad29015d0f959929ab HTTP/1.1" 200 None 2026-03-30 02:53:50,389 - lava_callback - ERROR - Error processing callback for node 69c742ad29015d0f959929ab: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:56566 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32876 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37226 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37232 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52344 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52350 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39202 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55092 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55014 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42192 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52896 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52906 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39026 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39036 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57584 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41776 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36254 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32994 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53158 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53160 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36354 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36366 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55818 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34572 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36178 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46628 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37562 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37572 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43436 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43438 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41190 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43082 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36414 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33814 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58874 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58888 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44048 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44050 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59452 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43078 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54678 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43556 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47466 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47474 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37458 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37470 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35116 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34888 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43510 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35282 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53066 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53082 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39690 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39696 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41264 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39306 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36934 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44776 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36102 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36118 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35044 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35054 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47548 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39606 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37792 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52902 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41226 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41238 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57742 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57746 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32844 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51328 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57416 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57006 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45664 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45668 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47988 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47994 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60284 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57492 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59130 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47286 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51920 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51926 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40082 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40084 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43142 - "GET /node/693f17d424745582e399b295%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:43158 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51296 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43660 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36396 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43722 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43734 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52378 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52380 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53090 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47086 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44072 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43222 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37890 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37898 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33026 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33036 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43780 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58704 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36832 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51026 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52176 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52190 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35468 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35474 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34006 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48370 - "GET / HTTP/1.0" 200 OK 2026-03-30 03:07:48,108 - lava_callback - INFO - Received callback for node 69c9e46b3c3aafdab4bea7d4 INFO: 172.20.0.1:48380 - "POST /node/69c9e46b3c3aafdab4bea7d4 HTTP/1.0" 202 Accepted 2026-03-30 03:07:48,116 - lava_callback - INFO - Starting async job submit for node 69c9e46b3c3aafdab4bea7d4 2026-03-30 03:07:48,159 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:07:48,178 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9e46b3c3aafdab4bea7d4 HTTP/1.1" 200 None 2026-03-30 03:07:48,179 - lava_callback - DEBUG - Processing node 69c9e46b3c3aafdab4bea7d4: baseline-nfs-arm64 2026-03-30 03:07:49,307 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 03:07:49,516 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 03:07:49,518 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9e46b3c3aafdab4bea7d4/log.txt.gz 2026-03-30 03:07:50,163 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 03:07:50,423 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 03:07:50,426 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9e46b3c3aafdab4bea7d4/lava-logs.yaml 2026-03-30 03:07:50,447 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 03:07:50,623 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 03:07:50,625 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9e46b3c3aafdab4bea7d4/lava_callback.json.gz DEBUG: Setup failed for 69c9e46b3c3aafdab4bea7d4. Transitting job node result: incomplete -> fail 2026-03-30 03:07:50,715 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:07:50,729 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9e46b3c3aafdab4bea7d4 HTTP/1.1" 200 None 2026-03-30 03:07:50,731 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:07:50,747 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 03:07:50,748 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:07:50,775 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9e46b3c3aafdab4bea7d4 HTTP/1.1" 200 None 2026-03-30 03:07:50,879 - lava_callback - INFO - Completed processing callback for node 69c9e46b3c3aafdab4bea7d4 INFO: 172.20.0.1:46844 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46856 - "GET /node/698313a0409235c29c0def49%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:47726 - "GET /node/693f59d30d8ce4018da8beaa%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:47732 - "GET / HTTP/1.0" 200 OK 2026-03-30 03:08:12,543 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:08:12,569 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:42888 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42898 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42418 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42422 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35898 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38926 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36028 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58866 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45126 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45132 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44918 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44920 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60532 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33638 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52554 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52542 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51610 - "GET /node/684be5a6cf8571aaed4e1faf%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:51620 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51632 - "GET /node/69a3c7766346680f7a28f384%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:51636 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41984 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41986 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53316 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45574 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58454 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55110 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33416 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33426 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42652 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42662 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34202 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59874 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60302 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38522 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34844 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34854 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37492 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37500 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43218 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44172 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37742 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39470 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59350 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59356 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51974 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51988 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36426 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45082 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44174 - "GET /node/698f59233013abef95cf7212%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:44184 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44188 - "GET /node/69411c1e459ff5fc64d2a90d%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:37210 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40600 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40608 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39328 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39334 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53400 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58102 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33306 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46082 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57386 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57398 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56074 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56082 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36392 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53182 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50244 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41892 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44720 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44722 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34896 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34908 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49588 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45660 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40110 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46312 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46316 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59242 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59246 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33168 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55326 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53390 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57514 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48616 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48626 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33710 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33726 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36588 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58912 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38434 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45064 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45076 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34466 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33840 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59306 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60762 - "GET /node/694c15612f7ca341fdeb0dea%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:60776 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52530 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39564 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39570 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58718 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58734 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42370 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59508 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44386 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32928 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59922 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59932 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49838 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49844 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60508 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51620 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50960 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34828 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38332 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38348 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38786 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38788 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48900 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56122 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52640 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38210 - "GET /node/6910a57d8b15debe9dd5297c%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:38220 - "GET /node/69903ab9985e4efca431da63%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:38230 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48126 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48142 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34898 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34910 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50140 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36748 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48960 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45812 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45816 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48002 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48018 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45196 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40292 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53794 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34312 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51772 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51774 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37804 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37812 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41196 - "GET / HTTP/1.0" 200 OK 2026-03-30 03:25:51,170 - lava_callback - INFO - Received callback for node 69c74a3729015d0f95992a59 INFO: 172.20.0.1:33370 - "POST /node/69c74a3729015d0f95992a59 HTTP/1.0" 202 Accepted 2026-03-30 03:25:51,176 - lava_callback - INFO - Starting async job submit for node 69c74a3729015d0f95992a59 2026-03-30 03:25:51,176 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:25:51,192 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c74a3729015d0f95992a59 HTTP/1.1" 200 None 2026-03-30 03:25:51,192 - lava_callback - DEBUG - Processing node 69c74a3729015d0f95992a59: baseline-arm64-baylibre 2026-03-30 03:25:51,193 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c74a3729015d0f95992a59 2026-03-30 03:25:51,193 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:25:51,218 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c74a3729015d0f95992a59 HTTP/1.1" 200 None 2026-03-30 03:25:51,219 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:25:51,232 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-30 03:25:51,233 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:25:51,253 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c74a3729015d0f95992a59 HTTP/1.1" 200 None 2026-03-30 03:25:51,254 - lava_callback - ERROR - Error processing callback for node 69c74a3729015d0f95992a59: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:33372 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51186 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59554 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59568 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49506 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49512 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57740 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49574 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60930 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44366 - "GET / HTTP/1.0" 200 OK 2026-03-30 03:27:11,234 - lava_callback - INFO - Received callback for node 69c74a8929015d0f95992a70 INFO: 172.20.0.1:35714 - "POST /node/69c74a8929015d0f95992a70 HTTP/1.0" 202 Accepted 2026-03-30 03:27:11,240 - lava_callback - INFO - Starting async job submit for node 69c74a8929015d0f95992a70 2026-03-30 03:27:11,241 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:27:11,266 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c74a8929015d0f95992a70 HTTP/1.1" 200 None 2026-03-30 03:27:11,267 - lava_callback - DEBUG - Processing node 69c74a8929015d0f95992a70: baseline-arm64-baylibre 2026-03-30 03:27:11,267 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c74a8929015d0f95992a70 2026-03-30 03:27:11,268 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:27:11,281 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c74a8929015d0f95992a70 HTTP/1.1" 200 None 2026-03-30 03:27:11,282 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:27:11,296 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-30 03:27:11,298 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:27:11,328 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c74a8929015d0f95992a70 HTTP/1.1" 200 None 2026-03-30 03:27:11,330 - lava_callback - ERROR - Error processing callback for node 69c74a8929015d0f95992a70: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35726 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35734 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59906 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59916 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59006 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39524 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56104 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58842 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42020 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42028 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45250 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45258 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46492 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51188 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50504 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33230 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37764 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37772 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57280 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57294 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41482 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39088 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44682 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50466 - "GET / HTTP/1.0" 200 OK 2026-03-30 03:30:11,349 - lava_callback - INFO - Received callback for node 69c74b2f29015d0f95992a96 INFO: 172.20.0.1:34398 - "POST /node/69c74b2f29015d0f95992a96 HTTP/1.0" 202 Accepted 2026-03-30 03:30:11,354 - lava_callback - INFO - Starting async job submit for node 69c74b2f29015d0f95992a96 2026-03-30 03:30:11,355 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:30:11,369 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c74b2f29015d0f95992a96 HTTP/1.1" 200 None 2026-03-30 03:30:11,370 - lava_callback - DEBUG - Processing node 69c74b2f29015d0f95992a96: baseline-arm64-baylibre 2026-03-30 03:30:11,371 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c74b2f29015d0f95992a96 2026-03-30 03:30:11,371 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:30:11,392 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c74b2f29015d0f95992a96 HTTP/1.1" 200 None 2026-03-30 03:30:11,394 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:30:11,408 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-30 03:30:11,409 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:30:11,443 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c74b2f29015d0f95992a96 HTTP/1.1" 200 None 2026-03-30 03:30:11,445 - lava_callback - ERROR - Error processing callback for node 69c74b2f29015d0f95992a96: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:34414 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34424 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53062 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53066 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47250 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40748 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38542 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36384 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50870 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50878 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33532 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33540 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38192 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49014 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59312 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42606 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58938 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58954 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38606 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38618 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41484 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42850 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35300 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41868 - "GET / HTTP/1.0" 200 OK 2026-03-30 03:33:12,037 - lava_callback - INFO - Received callback for node 69c74be429015d0f95992aa9 INFO: 172.20.0.1:38708 - "POST /node/69c74be429015d0f95992aa9 HTTP/1.0" 202 Accepted 2026-03-30 03:33:12,043 - lava_callback - INFO - Starting async job submit for node 69c74be429015d0f95992aa9 2026-03-30 03:33:12,044 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:33:12,058 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c74be429015d0f95992aa9 HTTP/1.1" 200 None 2026-03-30 03:33:12,059 - lava_callback - DEBUG - Processing node 69c74be429015d0f95992aa9: baseline-arm64-clabbe 2026-03-30 03:33:12,059 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c74be429015d0f95992aa9 2026-03-30 03:33:12,060 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:33:12,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c74be429015d0f95992aa9 HTTP/1.1" 200 None 2026-03-30 03:33:12,075 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:33:12,089 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-30 03:33:12,091 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:33:12,133 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c74be429015d0f95992aa9 HTTP/1.1" 200 None 2026-03-30 03:33:12,134 - lava_callback - ERROR - Error processing callback for node 69c74be429015d0f95992aa9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38710 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38714 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40856 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40866 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54358 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38266 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60512 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38138 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36088 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36100 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60942 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60956 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33906 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51216 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33282 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37896 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48722 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48738 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34276 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34286 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57522 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52490 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55114 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37292 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40312 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40318 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57820 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57832 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43078 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53808 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39126 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49338 - "GET / HTTP/1.0" 200 OK 2026-03-30 03:37:12,222 - lava_callback - INFO - Received callback for node 69c74cd429015d0f95992abb INFO: 172.20.0.1:33270 - "POST /node/69c74cd429015d0f95992abb HTTP/1.0" 202 Accepted 2026-03-30 03:37:12,228 - lava_callback - INFO - Starting async job submit for node 69c74cd429015d0f95992abb 2026-03-30 03:37:12,229 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:37:12,243 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c74cd429015d0f95992abb HTTP/1.1" 200 None 2026-03-30 03:37:12,244 - lava_callback - DEBUG - Processing node 69c74cd429015d0f95992abb: baseline-arm64-clabbe 2026-03-30 03:37:12,244 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c74cd429015d0f95992abb 2026-03-30 03:37:12,245 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:37:12,258 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c74cd429015d0f95992abb HTTP/1.1" 200 None 2026-03-30 03:37:12,259 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:37:12,283 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-30 03:37:12,285 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:37:12,306 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c74cd429015d0f95992abb HTTP/1.1" 200 None 2026-03-30 03:37:12,312 - lava_callback - ERROR - Error processing callback for node 69c74cd429015d0f95992abb: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:33282 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33298 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42708 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42724 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50882 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52736 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47006 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50560 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50570 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43236 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43240 - "GET / HTTP/1.0" 200 OK 2026-03-30 03:38:32,454 - lava_callback - INFO - Received callback for node 69c74d2629015d0f95992ac4 INFO: 172.20.0.1:58494 - "POST /node/69c74d2629015d0f95992ac4 HTTP/1.0" 202 Accepted 2026-03-30 03:38:32,459 - lava_callback - INFO - Starting async job submit for node 69c74d2629015d0f95992ac4 2026-03-30 03:38:32,460 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:38:32,475 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c74d2629015d0f95992ac4 HTTP/1.1" 200 None 2026-03-30 03:38:32,476 - lava_callback - DEBUG - Processing node 69c74d2629015d0f95992ac4: baseline-arm64-clabbe 2026-03-30 03:38:32,476 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c74d2629015d0f95992ac4 2026-03-30 03:38:32,477 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:38:32,490 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c74d2629015d0f95992ac4 HTTP/1.1" 200 None 2026-03-30 03:38:32,491 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:38:32,504 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-30 03:38:32,506 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:38:32,529 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c74d2629015d0f95992ac4 HTTP/1.1" 200 None 2026-03-30 03:38:32,534 - lava_callback - ERROR - Error processing callback for node 69c74d2629015d0f95992ac4: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:58496 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58072 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33944 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42406 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52566 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52576 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52200 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52212 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58824 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60732 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60056 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51306 - "GET / HTTP/1.0" 200 OK 2026-03-30 03:40:12,374 - lava_callback - INFO - Received callback for node 69c74d8829015d0f95992ad8 INFO: 172.20.0.1:33570 - "POST /node/69c74d8829015d0f95992ad8 HTTP/1.0" 202 Accepted 2026-03-30 03:40:12,379 - lava_callback - INFO - Starting async job submit for node 69c74d8829015d0f95992ad8 2026-03-30 03:40:12,380 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:40:12,399 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c74d8829015d0f95992ad8 HTTP/1.1" 200 None 2026-03-30 03:40:12,399 - lava_callback - DEBUG - Processing node 69c74d8829015d0f95992ad8: baseline-arm64-clabbe 2026-03-30 03:40:12,400 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c74d8829015d0f95992ad8 2026-03-30 03:40:12,400 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:40:12,414 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c74d8829015d0f95992ad8 HTTP/1.1" 200 None 2026-03-30 03:40:12,415 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:40:12,427 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-30 03:40:12,429 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:40:12,455 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c74d8829015d0f95992ad8 HTTP/1.1" 200 None 2026-03-30 03:40:12,457 - lava_callback - ERROR - Error processing callback for node 69c74d8829015d0f95992ad8: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:33572 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33588 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38762 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38772 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50730 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50374 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53818 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57618 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52872 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52886 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36246 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36260 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33600 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47914 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36310 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47698 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39110 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39120 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58376 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58386 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59952 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39864 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48592 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51606 - "GET / HTTP/1.0" 200 OK 2026-03-30 03:43:12,432 - lava_callback - INFO - Received callback for node 69c74e3c29015d0f95992ae7 INFO: 172.20.0.1:56272 - "POST /node/69c74e3c29015d0f95992ae7 HTTP/1.0" 202 Accepted 2026-03-30 03:43:12,437 - lava_callback - INFO - Starting async job submit for node 69c74e3c29015d0f95992ae7 2026-03-30 03:43:12,438 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:43:12,453 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c74e3c29015d0f95992ae7 HTTP/1.1" 200 None 2026-03-30 03:43:12,454 - lava_callback - DEBUG - Processing node 69c74e3c29015d0f95992ae7: baseline-arm64-clabbe 2026-03-30 03:43:12,454 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c74e3c29015d0f95992ae7 2026-03-30 03:43:12,455 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:43:12,474 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c74e3c29015d0f95992ae7 HTTP/1.1" 200 None 2026-03-30 03:43:12,475 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:43:12,506 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-30 03:43:12,507 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:43:12,528 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c74e3c29015d0f95992ae7 HTTP/1.1" 200 None 2026-03-30 03:43:12,535 - lava_callback - ERROR - Error processing callback for node 69c74e3c29015d0f95992ae7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:56282 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56288 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53962 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53966 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39162 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51530 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57658 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52780 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34972 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34982 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57070 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57074 - "GET / HTTP/1.0" 200 OK 2026-03-30 03:44:32,518 - lava_callback - INFO - Received callback for node 69c74e8f29015d0f95992aef INFO: 172.20.0.1:51294 - "POST /node/69c74e8f29015d0f95992aef HTTP/1.0" 202 Accepted 2026-03-30 03:44:32,524 - lava_callback - INFO - Starting async job submit for node 69c74e8f29015d0f95992aef 2026-03-30 03:44:32,525 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:44:32,540 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c74e8f29015d0f95992aef HTTP/1.1" 200 None 2026-03-30 03:44:32,541 - lava_callback - DEBUG - Processing node 69c74e8f29015d0f95992aef: baseline-arm64-clabbe 2026-03-30 03:44:32,541 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c74e8f29015d0f95992aef 2026-03-30 03:44:32,542 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:44:32,558 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c74e8f29015d0f95992aef HTTP/1.1" 200 None 2026-03-30 03:44:32,560 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:44:32,597 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c63cf0b98d4d3c02067dd7 HTTP/1.1" 200 None 2026-03-30 03:44:32,599 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 03:44:32,629 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c74e8f29015d0f95992aef HTTP/1.1" 200 None 2026-03-30 03:44:32,631 - lava_callback - ERROR - Error processing callback for node 69c74e8f29015d0f95992aef: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:51308 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46360 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39462 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42084 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60700 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60704 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37732 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37746 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50480 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33848 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48216 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41646 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46968 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46984 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48846 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48860 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56254 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44396 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60586 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41414 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58372 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58388 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38738 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38748 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44796 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39482 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48224 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57220 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58194 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58196 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37482 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37496 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59066 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58112 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56338 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46890 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53646 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53660 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54076 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54082 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37766 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60396 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52242 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44328 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58104 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58114 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35620 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35626 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36568 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60864 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43468 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49324 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48542 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48544 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43214 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43222 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59002 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40602 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54482 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45262 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60922 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60928 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44592 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44606 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56870 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45200 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51242 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35874 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35878 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50084 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50096 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60792 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58078 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55626 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42202 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33986 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34000 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48296 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48300 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35876 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36316 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57826 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42820 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58862 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58864 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47502 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47512 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59924 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34776 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52304 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59060 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41814 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41822 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51450 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51454 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51078 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56242 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37746 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55786 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55792 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36646 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34626 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60236 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60560 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56390 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48026 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48038 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44038 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44048 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47670 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37156 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56112 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39000 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48940 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48954 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43666 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43670 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50414 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32900 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60274 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41892 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41908 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44454 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44468 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60670 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39306 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36410 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33874 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37114 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37126 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41274 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41288 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43994 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36890 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40534 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38602 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47138 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47144 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33504 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33508 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54700 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39928 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37996 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47428 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55262 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55272 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35538 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35552 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50020 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45856 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55226 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38224 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48278 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48286 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43140 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43142 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38126 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41682 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48896 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34440 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34446 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54636 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54638 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58860 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40328 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44164 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34452 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41240 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41250 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54316 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54326 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43328 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40642 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51712 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36218 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44194 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44196 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40438 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40440 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58468 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35182 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57336 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59920 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40258 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40260 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37468 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37480 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52660 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51462 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48526 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35748 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59980 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59986 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46132 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46136 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49262 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36512 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59110 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44014 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45732 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45748 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35580 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35596 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57422 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47512 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55518 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39462 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55134 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55142 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41508 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41510 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57324 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49724 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40640 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33930 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37224 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37230 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56284 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56300 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55672 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38794 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56272 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59796 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47074 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47090 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56684 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56700 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38194 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50988 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41794 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38486 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58632 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58642 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49768 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49774 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42552 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47104 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36750 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33104 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51260 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51264 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46060 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46062 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54642 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52746 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45422 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42944 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43490 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43500 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60344 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60346 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55820 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36926 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41076 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55776 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45946 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45960 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47424 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47434 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49500 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41078 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58616 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53116 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53862 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53872 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45508 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45512 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46714 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55784 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40244 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54608 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38472 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38476 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57788 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57804 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57012 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59720 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50064 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36694 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36964 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36970 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39130 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39132 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36576 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47818 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47494 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33594 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33608 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33618 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55896 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55904 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58080 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57404 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36844 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59262 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42116 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42124 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43324 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34414 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49368 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35098 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40816 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52058 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52068 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33004 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33014 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50712 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56858 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58916 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38496 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48484 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48486 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45388 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45402 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48808 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53176 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46760 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52516 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44214 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44224 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59390 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59396 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55032 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40412 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56884 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39322 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39326 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57926 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57938 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53976 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49980 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37462 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40246 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59828 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59834 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54572 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54584 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54854 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50454 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48140 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58154 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58120 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58126 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36056 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36062 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51728 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37488 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50922 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48790 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56766 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56770 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44104 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44108 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47968 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42744 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34994 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51004 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54090 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54092 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59744 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59752 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43328 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37904 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52446 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34690 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43200 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43212 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60534 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60538 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44532 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55118 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48938 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37420 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48314 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50336 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56396 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55896 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46700 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40530 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46958 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46974 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52418 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52424 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44196 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34986 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37840 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54366 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40458 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40472 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56424 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56440 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45382 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41332 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50876 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55160 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60816 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60824 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38374 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38378 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56534 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58252 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46970 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43450 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51556 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51560 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48972 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48982 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42222 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44928 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36744 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50362 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52562 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52576 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43352 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43364 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55450 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58216 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42494 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53180 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48156 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48168 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54146 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54150 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53634 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43404 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45968 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48566 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45688 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45690 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45114 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45116 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39962 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51202 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49628 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53062 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36160 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36174 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38022 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38038 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36352 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49752 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44000 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34396 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60742 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60756 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46656 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46666 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43644 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51306 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44526 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58756 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53884 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53898 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48748 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48752 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43852 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36448 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57552 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43740 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46640 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46654 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40380 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40390 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54654 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49166 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40272 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33646 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33654 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57406 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57418 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56966 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38620 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47926 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45826 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42354 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42366 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48790 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51104 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48144 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55812 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47978 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35948 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35964 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47274 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47280 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35234 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56398 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49038 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57054 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42804 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35090 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35098 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54242 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51844 - "GET / HTTP/1.0" 200 OK 2026-03-30 04:47:56,095 - lava_callback - INFO - Received callback for node 69c75d7029015d0f95992b47 INFO: 172.20.0.1:47220 - "POST /node/69c75d7029015d0f95992b47 HTTP/1.0" 202 Accepted 2026-03-30 04:47:56,100 - lava_callback - INFO - Starting async job submit for node 69c75d7029015d0f95992b47 2026-03-30 04:47:56,101 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:47:56,125 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75d7029015d0f95992b47 HTTP/1.1" 200 None 2026-03-30 04:47:56,126 - lava_callback - DEBUG - Processing node 69c75d7029015d0f95992b47: baseline-arm-baylibre 2026-03-30 04:47:56,126 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c75d7029015d0f95992b47 2026-03-30 04:47:56,127 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:47:56,140 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75d7029015d0f95992b47 HTTP/1.1" 200 None 2026-03-30 04:47:56,142 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:47:56,154 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 04:47:56,156 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:47:56,177 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c75d7029015d0f95992b47 HTTP/1.1" 200 None 2026-03-30 04:47:56,178 - lava_callback - ERROR - Error processing callback for node 69c75d7029015d0f95992b47: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47236 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59746 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55268 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55280 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33684 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33690 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54366 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51016 - "GET / HTTP/1.0" 200 OK 2026-03-30 04:48:55,709 - lava_callback - INFO - Received callback for node 69c75dab29015d0f95992b56 INFO: 172.20.0.1:33052 - "POST /node/69c75dab29015d0f95992b56 HTTP/1.0" 202 Accepted 2026-03-30 04:48:55,714 - lava_callback - INFO - Starting async job submit for node 69c75dab29015d0f95992b56 2026-03-30 04:48:55,715 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:48:55,730 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75dab29015d0f95992b56 HTTP/1.1" 200 None 2026-03-30 04:48:55,731 - lava_callback - DEBUG - Processing node 69c75dab29015d0f95992b56: baseline-arm-baylibre 2026-03-30 04:48:55,731 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c75dab29015d0f95992b56 2026-03-30 04:48:55,732 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:48:55,748 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75dab29015d0f95992b56 HTTP/1.1" 200 None 2026-03-30 04:48:55,750 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:48:55,775 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 04:48:55,776 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:48:55,797 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c75dab29015d0f95992b56 HTTP/1.1" 200 None 2026-03-30 04:48:55,802 - lava_callback - ERROR - Error processing callback for node 69c75dab29015d0f95992b56: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:33064 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47286 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32934 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:32940 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37340 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37350 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38050 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35320 - "GET / HTTP/1.0" 200 OK 2026-03-30 04:49:55,862 - lava_callback - INFO - Received callback for node 69c75de729015d0f95992b64 INFO: 172.20.0.1:44208 - "POST /node/69c75de729015d0f95992b64 HTTP/1.0" 202 Accepted 2026-03-30 04:49:55,867 - lava_callback - INFO - Starting async job submit for node 69c75de729015d0f95992b64 2026-03-30 04:49:55,868 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:49:55,883 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75de729015d0f95992b64 HTTP/1.1" 200 None 2026-03-30 04:49:55,884 - lava_callback - DEBUG - Processing node 69c75de729015d0f95992b64: baseline-arm-baylibre 2026-03-30 04:49:55,884 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c75de729015d0f95992b64 2026-03-30 04:49:55,885 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:49:55,898 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75de729015d0f95992b64 HTTP/1.1" 200 None 2026-03-30 04:49:55,900 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:49:55,922 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 04:49:55,924 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:49:55,944 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c75de729015d0f95992b64 HTTP/1.1" 200 None 2026-03-30 04:49:55,950 - lava_callback - ERROR - Error processing callback for node 69c75de729015d0f95992b64: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44214 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49310 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55432 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55436 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56946 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56962 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34682 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46114 - "GET / HTTP/1.0" 200 OK 2026-03-30 04:50:55,854 - lava_callback - INFO - Received callback for node 69c75e2529015d0f95992b76 INFO: 172.20.0.1:38608 - "POST /node/69c75e2529015d0f95992b76 HTTP/1.0" 202 Accepted 2026-03-30 04:50:55,860 - lava_callback - INFO - Starting async job submit for node 69c75e2529015d0f95992b76 2026-03-30 04:50:55,860 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:50:55,876 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75e2529015d0f95992b76 HTTP/1.1" 200 None 2026-03-30 04:50:55,877 - lava_callback - DEBUG - Processing node 69c75e2529015d0f95992b76: baseline-arm-baylibre 2026-03-30 04:50:55,877 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c75e2529015d0f95992b76 2026-03-30 04:50:55,878 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:50:55,894 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75e2529015d0f95992b76 HTTP/1.1" 200 None 2026-03-30 04:50:55,896 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:50:55,910 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 04:50:55,912 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:50:55,932 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c75e2529015d0f95992b76 HTTP/1.1" 200 None 2026-03-30 04:50:55,938 - lava_callback - ERROR - Error processing callback for node 69c75e2529015d0f95992b76: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38624 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43228 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36804 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56600 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56612 - "GET / HTTP/1.0" 200 OK 2026-03-30 04:51:35,854 - lava_callback - INFO - Received callback for node 69c75e4829015d0f95992b82 INFO: 172.20.0.1:46602 - "POST /node/69c75e4829015d0f95992b82 HTTP/1.0" 202 Accepted 2026-03-30 04:51:35,859 - lava_callback - INFO - Starting async job submit for node 69c75e4829015d0f95992b82 2026-03-30 04:51:35,861 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:51:35,876 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75e4829015d0f95992b82 HTTP/1.1" 200 None 2026-03-30 04:51:35,876 - lava_callback - DEBUG - Processing node 69c75e4829015d0f95992b82: baseline-arm-baylibre 2026-03-30 04:51:35,877 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c75e4829015d0f95992b82 2026-03-30 04:51:35,877 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:51:35,891 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75e4829015d0f95992b82 HTTP/1.1" 200 None 2026-03-30 04:51:35,892 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:51:35,905 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 04:51:35,907 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:51:35,927 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c75e4829015d0f95992b82 HTTP/1.1" 200 None 2026-03-30 04:51:35,934 - lava_callback - ERROR - Error processing callback for node 69c75e4829015d0f95992b82: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46604 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56786 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37296 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52396 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55444 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-30 04:52:15,894 - lava_callback - INFO - Received callback for node 69c75e6d29015d0f95992b8e INFO: 172.20.0.1:55454 - "POST /node/69c75e6d29015d0f95992b8e HTTP/1.0" 202 Accepted 2026-03-30 04:52:15,900 - lava_callback - INFO - Starting async job submit for node 69c75e6d29015d0f95992b8e 2026-03-30 04:52:15,901 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:52:15,916 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75e6d29015d0f95992b8e HTTP/1.1" 200 None 2026-03-30 04:52:15,917 - lava_callback - DEBUG - Processing node 69c75e6d29015d0f95992b8e: baseline-arm-baylibre 2026-03-30 04:52:15,917 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c75e6d29015d0f95992b8e 2026-03-30 04:52:15,918 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:52:15,939 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75e6d29015d0f95992b8e HTTP/1.1" 200 None 2026-03-30 04:52:15,940 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:52:15,954 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 04:52:15,955 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:52:15,976 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c75e6d29015d0f95992b8e HTTP/1.1" 200 None 2026-03-30 04:52:15,977 - lava_callback - ERROR - Error processing callback for node 69c75e6d29015d0f95992b8e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59584 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59600 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51420 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46130 - "GET / HTTP/1.0" 200 OK 2026-03-30 04:52:55,976 - lava_callback - INFO - Received callback for node 69c75e9329015d0f95992b97 INFO: 172.20.0.1:45848 - "POST /node/69c75e9329015d0f95992b97 HTTP/1.0" 202 Accepted 2026-03-30 04:52:55,981 - lava_callback - INFO - Starting async job submit for node 69c75e9329015d0f95992b97 2026-03-30 04:52:55,982 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:52:55,997 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75e9329015d0f95992b97 HTTP/1.1" 200 None 2026-03-30 04:52:55,998 - lava_callback - DEBUG - Processing node 69c75e9329015d0f95992b97: baseline-arm-baylibre 2026-03-30 04:52:55,998 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c75e9329015d0f95992b97 2026-03-30 04:52:55,998 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:52:56,012 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75e9329015d0f95992b97 HTTP/1.1" 200 None 2026-03-30 04:52:56,013 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:52:56,036 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 04:52:56,038 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:52:56,059 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c75e9329015d0f95992b97 HTTP/1.1" 200 None 2026-03-30 04:52:56,064 - lava_callback - ERROR - Error processing callback for node 69c75e9329015d0f95992b97: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45858 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55570 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57482 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-30 04:53:15,962 - lava_callback - INFO - Received callback for node 69c75eb829015d0f95992ba6 INFO: 172.20.0.1:57494 - "POST /node/69c75eb829015d0f95992ba6 HTTP/1.0" 202 Accepted 2026-03-30 04:53:15,968 - lava_callback - INFO - Starting async job submit for node 69c75eb829015d0f95992ba6 2026-03-30 04:53:15,968 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:53:15,983 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75eb829015d0f95992ba6 HTTP/1.1" 200 None 2026-03-30 04:53:15,984 - lava_callback - DEBUG - Processing node 69c75eb829015d0f95992ba6: baseline-arm-baylibre 2026-03-30 04:53:15,984 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c75eb829015d0f95992ba6 2026-03-30 04:53:15,985 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:53:15,998 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75eb829015d0f95992ba6 HTTP/1.1" 200 None 2026-03-30 04:53:16,000 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:53:16,020 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 04:53:16,022 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:53:16,048 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c75eb829015d0f95992ba6 HTTP/1.1" 200 None 2026-03-30 04:53:16,049 - lava_callback - ERROR - Error processing callback for node 69c75eb829015d0f95992ba6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57496 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58790 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58792 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36724 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57888 - "GET / HTTP/1.0" 200 OK 2026-03-30 04:53:55,501 - lava_callback - INFO - Received callback for node 69c75ede29015d0f95992bb5 INFO: 172.20.0.1:43524 - "POST /node/69c75ede29015d0f95992bb5 HTTP/1.0" 202 Accepted 2026-03-30 04:53:55,507 - lava_callback - INFO - Starting async job submit for node 69c75ede29015d0f95992bb5 2026-03-30 04:53:55,508 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:53:55,522 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75ede29015d0f95992bb5 HTTP/1.1" 200 None 2026-03-30 04:53:55,523 - lava_callback - DEBUG - Processing node 69c75ede29015d0f95992bb5: baseline-arm-clabbe 2026-03-30 04:53:55,523 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c75ede29015d0f95992bb5 2026-03-30 04:53:55,524 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:53:55,537 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75ede29015d0f95992bb5 HTTP/1.1" 200 None 2026-03-30 04:53:55,538 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:53:55,551 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 04:53:55,553 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:53:55,573 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c75ede29015d0f95992bb5 HTTP/1.1" 200 None 2026-03-30 04:53:55,574 - lava_callback - ERROR - Error processing callback for node 69c75ede29015d0f95992bb5: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43526 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59100 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50056 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50068 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33006 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33020 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59016 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40376 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52656 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52746 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-30 04:55:15,420 - lava_callback - INFO - Received callback for node 69c75f2529015d0f95992bd1 INFO: 172.20.0.1:52762 - "POST /node/69c75f2529015d0f95992bd1 HTTP/1.0" 202 Accepted 2026-03-30 04:55:15,426 - lava_callback - INFO - Starting async job submit for node 69c75f2529015d0f95992bd1 2026-03-30 04:55:15,427 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:55:15,442 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75f2529015d0f95992bd1 HTTP/1.1" 200 None 2026-03-30 04:55:15,443 - lava_callback - DEBUG - Processing node 69c75f2529015d0f95992bd1: baseline-arm-clabbe 2026-03-30 04:55:15,443 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c75f2529015d0f95992bd1 2026-03-30 04:55:15,444 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:55:15,457 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75f2529015d0f95992bd1 HTTP/1.1" 200 None 2026-03-30 04:55:15,458 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:55:15,472 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 04:55:15,473 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:55:15,537 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c75f2529015d0f95992bd1 HTTP/1.1" 200 None 2026-03-30 04:55:15,539 - lava_callback - ERROR - Error processing callback for node 69c75f2529015d0f95992bd1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:52776 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39166 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39172 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53688 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52420 - "GET / HTTP/1.0" 200 OK 2026-03-30 04:55:55,559 - lava_callback - INFO - Received callback for node 69c75f4a29015d0f95992bde INFO: 172.20.0.1:44012 - "POST /node/69c75f4a29015d0f95992bde HTTP/1.0" 202 Accepted 2026-03-30 04:55:55,564 - lava_callback - INFO - Starting async job submit for node 69c75f4a29015d0f95992bde 2026-03-30 04:55:55,565 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:55:55,580 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75f4a29015d0f95992bde HTTP/1.1" 200 None 2026-03-30 04:55:55,581 - lava_callback - DEBUG - Processing node 69c75f4a29015d0f95992bde: baseline-arm-clabbe 2026-03-30 04:55:55,581 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c75f4a29015d0f95992bde 2026-03-30 04:55:55,582 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:55:55,623 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75f4a29015d0f95992bde HTTP/1.1" 200 None 2026-03-30 04:55:55,624 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:55:55,649 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 04:55:55,650 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:55:55,671 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c75f4a29015d0f95992bde HTTP/1.1" 200 None 2026-03-30 04:55:55,677 - lava_callback - ERROR - Error processing callback for node 69c75f4a29015d0f95992bde: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44018 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57418 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60530 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60542 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49150 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49166 - "GET / HTTP/1.0" 200 OK 2026-03-30 04:56:35,626 - lava_callback - INFO - Received callback for node 69c75f7029015d0f95992bee INFO: 172.20.0.1:36712 - "POST /node/69c75f7029015d0f95992bee HTTP/1.0" 202 Accepted 2026-03-30 04:56:35,632 - lava_callback - INFO - Starting async job submit for node 69c75f7029015d0f95992bee 2026-03-30 04:56:35,633 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:56:35,671 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75f7029015d0f95992bee HTTP/1.1" 200 None 2026-03-30 04:56:35,671 - lava_callback - DEBUG - Processing node 69c75f7029015d0f95992bee: baseline-arm-clabbe 2026-03-30 04:56:35,672 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c75f7029015d0f95992bee 2026-03-30 04:56:35,672 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:56:35,685 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75f7029015d0f95992bee HTTP/1.1" 200 None 2026-03-30 04:56:35,686 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:56:35,700 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 04:56:35,701 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:56:35,722 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c75f7029015d0f95992bee HTTP/1.1" 200 None 2026-03-30 04:56:35,731 - lava_callback - ERROR - Error processing callback for node 69c75f7029015d0f95992bee: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:36728 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43126 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33004 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41890 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35996 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36000 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41418 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41420 - "GET / HTTP/1.0" 200 OK 2026-03-30 04:57:35,649 - lava_callback - INFO - Received callback for node 69c75fb929015d0f95992c06 INFO: 172.20.0.1:47212 - "POST /node/69c75fb929015d0f95992c06 HTTP/1.0" 202 Accepted 2026-03-30 04:57:35,655 - lava_callback - INFO - Starting async job submit for node 69c75fb929015d0f95992c06 2026-03-30 04:57:35,656 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:57:35,670 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75fb929015d0f95992c06 HTTP/1.1" 200 None 2026-03-30 04:57:35,671 - lava_callback - DEBUG - Processing node 69c75fb929015d0f95992c06: baseline-arm-clabbe 2026-03-30 04:57:35,671 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c75fb929015d0f95992c06 2026-03-30 04:57:35,671 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:57:35,685 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75fb929015d0f95992c06 HTTP/1.1" 200 None 2026-03-30 04:57:35,686 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:57:35,700 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 04:57:35,702 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:57:35,724 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c75fb929015d0f95992c06 HTTP/1.1" 200 None 2026-03-30 04:57:35,736 - lava_callback - ERROR - Error processing callback for node 69c75fb929015d0f95992c06: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47226 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46094 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39370 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54304 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33242 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-30 04:58:15,645 - lava_callback - INFO - Received callback for node 69c75fde29015d0f95992c11 INFO: 172.20.0.1:33248 - "POST /node/69c75fde29015d0f95992c11 HTTP/1.0" 202 Accepted 2026-03-30 04:58:15,650 - lava_callback - INFO - Starting async job submit for node 69c75fde29015d0f95992c11 2026-03-30 04:58:15,651 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:58:15,666 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75fde29015d0f95992c11 HTTP/1.1" 200 None 2026-03-30 04:58:15,667 - lava_callback - DEBUG - Processing node 69c75fde29015d0f95992c11: baseline-arm-clabbe 2026-03-30 04:58:15,667 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c75fde29015d0f95992c11 2026-03-30 04:58:15,668 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:58:15,682 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c75fde29015d0f95992c11 HTTP/1.1" 200 None 2026-03-30 04:58:15,684 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:58:15,697 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 04:58:15,698 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:58:15,718 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c75fde29015d0f95992c11 HTTP/1.1" 200 None 2026-03-30 04:58:15,725 - lava_callback - ERROR - Error processing callback for node 69c75fde29015d0f95992c11: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:33256 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54468 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54484 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56568 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57096 - "GET / HTTP/1.0" 200 OK 2026-03-30 04:58:55,716 - lava_callback - INFO - Received callback for node 69c7600429015d0f95992c21 INFO: 172.20.0.1:41348 - "POST /node/69c7600429015d0f95992c21 HTTP/1.0" 202 Accepted 2026-03-30 04:58:55,722 - lava_callback - INFO - Starting async job submit for node 69c7600429015d0f95992c21 2026-03-30 04:58:55,723 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:58:55,743 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7600429015d0f95992c21 HTTP/1.1" 200 None 2026-03-30 04:58:55,743 - lava_callback - DEBUG - Processing node 69c7600429015d0f95992c21: baseline-arm-clabbe 2026-03-30 04:58:55,744 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7600429015d0f95992c21 2026-03-30 04:58:55,744 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:58:55,759 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7600429015d0f95992c21 HTTP/1.1" 200 None 2026-03-30 04:58:55,760 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:58:55,773 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 04:58:55,774 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:58:55,795 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7600429015d0f95992c21 HTTP/1.1" 200 None 2026-03-30 04:58:55,801 - lava_callback - ERROR - Error processing callback for node 69c7600429015d0f95992c21: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41364 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58494 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44196 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44208 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58988 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58994 - "GET / HTTP/1.0" 200 OK 2026-03-30 04:59:35,699 - lava_callback - INFO - Received callback for node 69c7602929015d0f95992c2f INFO: 172.20.0.1:49772 - "POST /node/69c7602929015d0f95992c2f HTTP/1.0" 202 Accepted 2026-03-30 04:59:35,704 - lava_callback - INFO - Starting async job submit for node 69c7602929015d0f95992c2f 2026-03-30 04:59:35,705 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:59:35,720 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7602929015d0f95992c2f HTTP/1.1" 200 None 2026-03-30 04:59:35,720 - lava_callback - DEBUG - Processing node 69c7602929015d0f95992c2f: baseline-arm-clabbe 2026-03-30 04:59:35,721 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7602929015d0f95992c2f 2026-03-30 04:59:35,721 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:59:35,735 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7602929015d0f95992c2f HTTP/1.1" 200 None 2026-03-30 04:59:35,737 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:59:35,750 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 04:59:35,751 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 04:59:35,784 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7602929015d0f95992c2f HTTP/1.1" 200 None 2026-03-30 04:59:35,786 - lava_callback - ERROR - Error processing callback for node 69c7602929015d0f95992c2f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49786 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47382 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58074 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52262 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32836 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-30 05:00:15,907 - lava_callback - INFO - Received callback for node 69c7604e29015d0f95992c3c INFO: 172.20.0.1:32842 - "POST /node/69c7604e29015d0f95992c3c HTTP/1.0" 202 Accepted 2026-03-30 05:00:15,912 - lava_callback - INFO - Starting async job submit for node 69c7604e29015d0f95992c3c 2026-03-30 05:00:15,913 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:00:15,927 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7604e29015d0f95992c3c HTTP/1.1" 200 None 2026-03-30 05:00:15,928 - lava_callback - DEBUG - Processing node 69c7604e29015d0f95992c3c: baseline-arm-clabbe 2026-03-30 05:00:15,928 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7604e29015d0f95992c3c 2026-03-30 05:00:15,929 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:00:15,942 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7604e29015d0f95992c3c HTTP/1.1" 200 None 2026-03-30 05:00:15,943 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:00:15,956 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 05:00:15,957 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:00:15,979 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7604e29015d0f95992c3c HTTP/1.1" 200 None 2026-03-30 05:00:15,992 - lava_callback - ERROR - Error processing callback for node 69c7604e29015d0f95992c3c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:32846 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52582 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52596 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48506 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49646 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:00:55,796 - lava_callback - INFO - Received callback for node 69c7607429015d0f95992c48 INFO: 172.20.0.1:42390 - "POST /node/69c7607429015d0f95992c48 HTTP/1.0" 202 Accepted 2026-03-30 05:00:55,802 - lava_callback - INFO - Starting async job submit for node 69c7607429015d0f95992c48 2026-03-30 05:00:55,802 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:00:55,822 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7607429015d0f95992c48 HTTP/1.1" 200 None 2026-03-30 05:00:55,823 - lava_callback - DEBUG - Processing node 69c7607429015d0f95992c48: baseline-arm-clabbe 2026-03-30 05:00:55,824 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7607429015d0f95992c48 2026-03-30 05:00:55,824 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:00:55,844 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7607429015d0f95992c48 HTTP/1.1" 200 None 2026-03-30 05:00:55,846 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:00:55,865 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 05:00:55,867 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:00:55,889 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7607429015d0f95992c48 HTTP/1.1" 200 None 2026-03-30 05:00:55,904 - lava_callback - ERROR - Error processing callback for node 69c7607429015d0f95992c48: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:42402 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48786 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53164 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-30 05:01:15,826 - lava_callback - INFO - Received callback for node 69c7609929015d0f95992c59 INFO: 172.20.0.1:53168 - "POST /node/69c7609929015d0f95992c59 HTTP/1.0" 202 Accepted 2026-03-30 05:01:15,835 - lava_callback - INFO - Starting async job submit for node 69c7609929015d0f95992c59 2026-03-30 05:01:15,836 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:01:15,854 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7609929015d0f95992c59 HTTP/1.1" 200 None 2026-03-30 05:01:15,855 - lava_callback - DEBUG - Processing node 69c7609929015d0f95992c59: baseline-arm-clabbe 2026-03-30 05:01:15,855 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7609929015d0f95992c59 2026-03-30 05:01:15,856 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:01:15,869 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7609929015d0f95992c59 HTTP/1.1" 200 None 2026-03-30 05:01:15,870 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:01:15,883 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 05:01:15,884 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:01:15,904 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7609929015d0f95992c59 HTTP/1.1" 200 None 2026-03-30 05:01:15,911 - lava_callback - ERROR - Error processing callback for node 69c7609929015d0f95992c59: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:53178 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51988 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52000 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49652 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37166 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:01:55,819 - lava_callback - INFO - Received callback for node 69c760be29015d0f95992c69 INFO: 172.20.0.1:55952 - "POST /node/69c760be29015d0f95992c69 HTTP/1.0" 202 Accepted 2026-03-30 05:01:55,825 - lava_callback - INFO - Starting async job submit for node 69c760be29015d0f95992c69 2026-03-30 05:01:55,826 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:01:55,840 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c760be29015d0f95992c69 HTTP/1.1" 200 None 2026-03-30 05:01:55,841 - lava_callback - DEBUG - Processing node 69c760be29015d0f95992c69: baseline-arm-clabbe 2026-03-30 05:01:55,842 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c760be29015d0f95992c69 2026-03-30 05:01:55,842 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:01:55,855 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c760be29015d0f95992c69 HTTP/1.1" 200 None 2026-03-30 05:01:55,857 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:01:55,870 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 05:01:55,871 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:01:55,897 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c760be29015d0f95992c69 HTTP/1.1" 200 None 2026-03-30 05:01:55,902 - lava_callback - ERROR - Error processing callback for node 69c760be29015d0f95992c69: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55964 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50558 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54328 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54342 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52026 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52038 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:02:35,890 - lava_callback - INFO - Received callback for node 69c760e329015d0f95992c72 INFO: 172.20.0.1:44954 - "POST /node/69c760e329015d0f95992c72 HTTP/1.0" 202 Accepted 2026-03-30 05:02:35,896 - lava_callback - INFO - Starting async job submit for node 69c760e329015d0f95992c72 2026-03-30 05:02:35,897 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:02:35,911 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c760e329015d0f95992c72 HTTP/1.1" 200 None 2026-03-30 05:02:35,912 - lava_callback - DEBUG - Processing node 69c760e329015d0f95992c72: baseline-arm-clabbe 2026-03-30 05:02:35,912 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c760e329015d0f95992c72 2026-03-30 05:02:35,913 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:02:35,926 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c760e329015d0f95992c72 HTTP/1.1" 200 None 2026-03-30 05:02:35,927 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:02:35,941 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 05:02:35,942 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:02:35,972 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c760e329015d0f95992c72 HTTP/1.1" 200 None 2026-03-30 05:02:35,973 - lava_callback - ERROR - Error processing callback for node 69c760e329015d0f95992c72: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44960 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41614 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47204 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37360 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43662 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43672 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46814 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46816 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43456 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37766 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60160 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48060 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43428 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43430 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46610 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46624 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57366 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57560 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58274 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45452 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41384 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41400 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46648 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46656 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56534 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51938 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59930 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41962 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41976 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43054 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43066 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47368 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59534 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44040 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58920 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58926 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40542 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40546 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55302 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47404 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43950 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42702 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47838 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47846 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51906 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51912 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39830 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37462 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43502 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:08:56,773 - lava_callback - INFO - Received callback for node 69c7625629015d0f95992cb5 INFO: 172.20.0.1:43510 - "POST /node/69c7625629015d0f95992cb5 HTTP/1.0" 202 Accepted 2026-03-30 05:08:56,779 - lava_callback - INFO - Starting async job submit for node 69c7625629015d0f95992cb5 2026-03-30 05:08:56,780 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:08:56,794 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7625629015d0f95992cb5 HTTP/1.1" 200 None 2026-03-30 05:08:56,795 - lava_callback - DEBUG - Processing node 69c7625629015d0f95992cb5: baseline-arm64-baylibre 2026-03-30 05:08:56,795 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7625629015d0f95992cb5 2026-03-30 05:08:56,796 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:08:56,809 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7625629015d0f95992cb5 HTTP/1.1" 200 None 2026-03-30 05:08:56,811 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:08:56,825 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6ae1602375b4506df899e HTTP/1.1" 200 None 2026-03-30 05:08:56,826 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:08:56,848 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7625629015d0f95992cb5 HTTP/1.1" 200 None 2026-03-30 05:08:56,850 - lava_callback - ERROR - Error processing callback for node 69c7625629015d0f95992cb5: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54334 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34916 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34930 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55844 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55860 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43710 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37430 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49268 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36222 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59252 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59254 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56062 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56068 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40072 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45724 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54998 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42916 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33098 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33112 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40488 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40504 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41768 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:11:36,898 - lava_callback - INFO - Received callback for node 69c762fd29015d0f95992ccd INFO: 172.20.0.1:41784 - "POST /node/69c762fd29015d0f95992ccd HTTP/1.0" 202 Accepted 2026-03-30 05:11:36,904 - lava_callback - INFO - Starting async job submit for node 69c762fd29015d0f95992ccd 2026-03-30 05:11:36,904 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:11:36,919 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c762fd29015d0f95992ccd HTTP/1.1" 200 None 2026-03-30 05:11:36,920 - lava_callback - DEBUG - Processing node 69c762fd29015d0f95992ccd: baseline-arm64-baylibre 2026-03-30 05:11:36,920 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c762fd29015d0f95992ccd 2026-03-30 05:11:36,921 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:11:36,934 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c762fd29015d0f95992ccd HTTP/1.1" 200 None 2026-03-30 05:11:36,936 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:11:36,950 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6ae1602375b4506df899e HTTP/1.1" 200 None 2026-03-30 05:11:36,952 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:11:36,986 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c762fd29015d0f95992ccd HTTP/1.1" 200 None 2026-03-30 05:11:36,995 - lava_callback - ERROR - Error processing callback for node 69c762fd29015d0f95992ccd: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41788 - "GET /node/69861d47177ca0f097bd1749%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:50762 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47008 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59432 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37402 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37406 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39522 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39538 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60356 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59376 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56966 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60300 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57704 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57712 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42990 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42994 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32790 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44170 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38432 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49096 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56452 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56454 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:14:17,035 - lava_callback - INFO - Received callback for node 69c763a029015d0f95992ce5 INFO: 172.20.0.1:56456 - "POST /node/69c763a029015d0f95992ce5 HTTP/1.0" 202 Accepted 2026-03-30 05:14:17,041 - lava_callback - INFO - Starting async job submit for node 69c763a029015d0f95992ce5 2026-03-30 05:14:17,041 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:14:17,061 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c763a029015d0f95992ce5 HTTP/1.1" 200 None 2026-03-30 05:14:17,062 - lava_callback - DEBUG - Processing node 69c763a029015d0f95992ce5: baseline-arm64-baylibre 2026-03-30 05:14:17,062 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c763a029015d0f95992ce5 2026-03-30 05:14:17,063 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:14:17,078 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c763a029015d0f95992ce5 HTTP/1.1" 200 None 2026-03-30 05:14:17,079 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:14:17,105 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 05:14:17,107 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:14:17,158 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c763a029015d0f95992ce5 HTTP/1.1" 200 None 2026-03-30 05:14:17,160 - lava_callback - ERROR - Error processing callback for node 69c763a029015d0f95992ce5: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:34720 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34722 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43898 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54132 - "GET /node/6986227b177ca0f097bd1935%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:54148 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44976 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37648 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34016 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34028 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59096 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59100 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43786 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:15:37,130 - lava_callback - INFO - Received callback for node 69c763f529015d0f95992cf4 INFO: 172.20.0.1:43788 - "POST /node/69c763f529015d0f95992cf4 HTTP/1.0" 202 Accepted 2026-03-30 05:15:37,136 - lava_callback - INFO - Starting async job submit for node 69c763f529015d0f95992cf4 2026-03-30 05:15:37,137 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:15:37,150 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c763f529015d0f95992cf4 HTTP/1.1" 200 None 2026-03-30 05:15:37,151 - lava_callback - DEBUG - Processing node 69c763f529015d0f95992cf4: baseline-arm64-baylibre 2026-03-30 05:15:37,151 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c763f529015d0f95992cf4 2026-03-30 05:15:37,152 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:15:37,165 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c763f529015d0f95992cf4 HTTP/1.1" 200 None 2026-03-30 05:15:37,166 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:15:37,187 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6ae1602375b4506df899e HTTP/1.1" 200 None 2026-03-30 05:15:37,189 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:15:37,216 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c763f529015d0f95992cf4 HTTP/1.1" 200 None 2026-03-30 05:15:37,221 - lava_callback - ERROR - Error processing callback for node 69c763f529015d0f95992cf4: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59524 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45704 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52466 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46388 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46396 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41544 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41546 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45582 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50156 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33108 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:16:57,153 - lava_callback - INFO - Received callback for node 69c7644629015d0f95992d08 INFO: 172.20.0.1:33116 - "POST /node/69c7644629015d0f95992d08 HTTP/1.0" 202 Accepted 2026-03-30 05:16:57,158 - lava_callback - INFO - Starting async job submit for node 69c7644629015d0f95992d08 2026-03-30 05:16:57,159 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:16:57,173 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7644629015d0f95992d08 HTTP/1.1" 200 None 2026-03-30 05:16:57,174 - lava_callback - DEBUG - Processing node 69c7644629015d0f95992d08: baseline-arm64-baylibre 2026-03-30 05:16:57,175 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7644629015d0f95992d08 2026-03-30 05:16:57,175 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:16:57,199 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7644629015d0f95992d08 HTTP/1.1" 200 None 2026-03-30 05:16:57,200 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:16:57,213 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 05:16:57,215 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:16:57,235 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7644629015d0f95992d08 HTTP/1.1" 200 None 2026-03-30 05:16:57,236 - lava_callback - ERROR - Error processing callback for node 69c7644629015d0f95992d08: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:37806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50920 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50930 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43756 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43770 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55306 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46782 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35908 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47232 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53826 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53832 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56812 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56814 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55002 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:18:37,275 - lava_callback - INFO - Received callback for node 69c7649b29015d0f95992d16 INFO: 172.20.0.1:55012 - "POST /node/69c7649b29015d0f95992d16 HTTP/1.0" 202 Accepted 2026-03-30 05:18:37,281 - lava_callback - INFO - Starting async job submit for node 69c7649b29015d0f95992d16 2026-03-30 05:18:37,281 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:18:37,296 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7649b29015d0f95992d16 HTTP/1.1" 200 None 2026-03-30 05:18:37,297 - lava_callback - DEBUG - Processing node 69c7649b29015d0f95992d16: baseline-arm64-baylibre 2026-03-30 05:18:37,297 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7649b29015d0f95992d16 2026-03-30 05:18:37,298 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:18:37,311 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7649b29015d0f95992d16 HTTP/1.1" 200 None 2026-03-30 05:18:37,312 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:18:37,325 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6ae1602375b4506df899e HTTP/1.1" 200 None 2026-03-30 05:18:37,326 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:18:37,346 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7649b29015d0f95992d16 HTTP/1.1" 200 None 2026-03-30 05:18:37,351 - lava_callback - ERROR - Error processing callback for node 69c7649b29015d0f95992d16: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:56582 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49612 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35566 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50800 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50802 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41342 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41356 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43724 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53048 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60030 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:19:57,356 - lava_callback - INFO - Received callback for node 69c764f329015d0f95992d20 INFO: 172.20.0.1:60036 - "POST /node/69c764f329015d0f95992d20 HTTP/1.0" 202 Accepted 2026-03-30 05:19:57,362 - lava_callback - INFO - Starting async job submit for node 69c764f329015d0f95992d20 2026-03-30 05:19:57,363 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:19:57,377 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c764f329015d0f95992d20 HTTP/1.1" 200 None 2026-03-30 05:19:57,378 - lava_callback - DEBUG - Processing node 69c764f329015d0f95992d20: baseline-arm64-baylibre 2026-03-30 05:19:57,378 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c764f329015d0f95992d20 2026-03-30 05:19:57,379 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:19:57,392 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c764f329015d0f95992d20 HTTP/1.1" 200 None 2026-03-30 05:19:57,394 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:19:57,407 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 05:19:57,409 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:19:57,431 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c764f329015d0f95992d20 HTTP/1.1" 200 None 2026-03-30 05:19:57,432 - lava_callback - ERROR - Error processing callback for node 69c764f329015d0f95992d20: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49236 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57392 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57400 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33470 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33476 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49680 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35322 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42278 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53718 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58468 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58472 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:21:16,714 - lava_callback - INFO - Received callback for node 69c7653e29015d0f95992d2d INFO: 172.20.0.1:58486 - "POST /node/69c7653e29015d0f95992d2d HTTP/1.0" 202 Accepted 2026-03-30 05:21:16,720 - lava_callback - INFO - Starting async job submit for node 69c7653e29015d0f95992d2d 2026-03-30 05:21:16,721 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:21:16,736 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7653e29015d0f95992d2d HTTP/1.1" 200 None 2026-03-30 05:21:16,737 - lava_callback - DEBUG - Processing node 69c7653e29015d0f95992d2d: baseline-arm-clabbe 2026-03-30 05:21:16,737 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7653e29015d0f95992d2d 2026-03-30 05:21:16,738 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:21:16,753 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7653e29015d0f95992d2d HTTP/1.1" 200 None 2026-03-30 05:21:16,755 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:21:16,769 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 05:21:16,770 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:21:16,790 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7653e29015d0f95992d2d HTTP/1.1" 200 None 2026-03-30 05:21:16,796 - lava_callback - ERROR - Error processing callback for node 69c7653e29015d0f95992d2d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:51402 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51410 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51418 - "GET /node/694940ecebaed71d438084ef%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:55050 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43654 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52540 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:21:56,665 - lava_callback - INFO - Received callback for node 69c7656529015d0f95992d39 INFO: 172.20.0.1:52544 - "POST /node/69c7656529015d0f95992d39 HTTP/1.0" 202 Accepted 2026-03-30 05:21:56,671 - lava_callback - INFO - Starting async job submit for node 69c7656529015d0f95992d39 2026-03-30 05:21:56,672 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:21:56,686 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7656529015d0f95992d39 HTTP/1.1" 200 None 2026-03-30 05:21:56,687 - lava_callback - DEBUG - Processing node 69c7656529015d0f95992d39: baseline-arm-clabbe 2026-03-30 05:21:56,687 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7656529015d0f95992d39 2026-03-30 05:21:56,688 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:21:56,701 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7656529015d0f95992d39 HTTP/1.1" 200 None 2026-03-30 05:21:56,703 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:21:56,716 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ac8944faaf0e7f5e408 HTTP/1.1" 200 None 2026-03-30 05:21:56,718 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:21:56,738 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7656529015d0f95992d39 HTTP/1.1" 200 None 2026-03-30 05:21:56,739 - lava_callback - ERROR - Error processing callback for node 69c7656529015d0f95992d39: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38856 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57296 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57298 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53574 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53578 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57540 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:22:37,536 - lava_callback - INFO - Received callback for node 69c7659329015d0f95992d46 INFO: 172.20.0.1:57552 - "POST /node/69c7659329015d0f95992d46 HTTP/1.0" 202 Accepted 2026-03-30 05:22:37,542 - lava_callback - INFO - Starting async job submit for node 69c7659329015d0f95992d46 2026-03-30 05:22:37,543 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:22:37,558 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7659329015d0f95992d46 HTTP/1.1" 200 None 2026-03-30 05:22:37,558 - lava_callback - DEBUG - Processing node 69c7659329015d0f95992d46: baseline-arm64-baylibre 2026-03-30 05:22:37,559 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7659329015d0f95992d46 2026-03-30 05:22:37,559 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:22:37,576 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7659329015d0f95992d46 HTTP/1.1" 200 None 2026-03-30 05:22:37,578 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:22:37,617 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 05:22:37,619 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:22:37,639 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7659329015d0f95992d46 HTTP/1.1" 200 None 2026-03-30 05:22:37,644 - lava_callback - ERROR - Error processing callback for node 69c7659329015d0f95992d46: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49710 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39364 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56890 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40708 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40722 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47554 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47560 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45764 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34210 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58332 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57136 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54672 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54678 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:24:16,876 - lava_callback - INFO - Received callback for node 69c765eb29015d0f95992d5e INFO: 172.20.0.1:54694 - "POST /node/69c765eb29015d0f95992d5e HTTP/1.0" 202 Accepted 2026-03-30 05:24:16,882 - lava_callback - INFO - Starting async job submit for node 69c765eb29015d0f95992d5e 2026-03-30 05:24:16,883 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:24:16,898 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c765eb29015d0f95992d5e HTTP/1.1" 200 None 2026-03-30 05:24:16,898 - lava_callback - DEBUG - Processing node 69c765eb29015d0f95992d5e: baseline-arm64-clabbe 2026-03-30 05:24:16,899 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c765eb29015d0f95992d5e 2026-03-30 05:24:16,899 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:24:16,924 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c765eb29015d0f95992d5e HTTP/1.1" 200 None 2026-03-30 05:24:16,926 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:24:16,940 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6ae1602375b4506df899e HTTP/1.1" 200 None 2026-03-30 05:24:16,941 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:24:16,972 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c765eb29015d0f95992d5e HTTP/1.1" 200 None 2026-03-30 05:24:16,983 - lava_callback - ERROR - Error processing callback for node 69c765eb29015d0f95992d5e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:53664 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53680 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53574 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48188 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48192 - "GET /node/69b7e1af00b80c8ffdcb8ef9%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:48196 - "GET /node/6995c66c445a8f9facd5d3ce%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:34974 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53324 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33126 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33130 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38670 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38678 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56124 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:25:36,867 - lava_callback - INFO - Received callback for node 69c7663d29015d0f95992d67 INFO: 172.20.0.1:56132 - "POST /node/69c7663d29015d0f95992d67 HTTP/1.0" 202 Accepted 2026-03-30 05:25:36,873 - lava_callback - INFO - Starting async job submit for node 69c7663d29015d0f95992d67 2026-03-30 05:25:36,874 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:25:36,888 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7663d29015d0f95992d67 HTTP/1.1" 200 None 2026-03-30 05:25:36,889 - lava_callback - DEBUG - Processing node 69c7663d29015d0f95992d67: baseline-arm64-clabbe 2026-03-30 05:25:36,889 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7663d29015d0f95992d67 2026-03-30 05:25:36,890 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:25:36,903 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7663d29015d0f95992d67 HTTP/1.1" 200 None 2026-03-30 05:25:36,904 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:25:36,917 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6ae1602375b4506df899e HTTP/1.1" 200 None 2026-03-30 05:25:36,918 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:25:36,967 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7663d29015d0f95992d67 HTTP/1.1" 200 None 2026-03-30 05:25:36,972 - lava_callback - ERROR - Error processing callback for node 69c7663d29015d0f95992d67: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:36058 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47254 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41352 - "GET /node/6985d39471989fb141499d16%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:41366 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50338 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50348 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40048 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40054 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53436 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42296 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49202 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:26:56,900 - lava_callback - INFO - Received callback for node 69c7669129015d0f95992d76 INFO: 172.20.0.1:49216 - "POST /node/69c7669129015d0f95992d76 HTTP/1.0" 202 Accepted 2026-03-30 05:26:56,906 - lava_callback - INFO - Starting async job submit for node 69c7669129015d0f95992d76 2026-03-30 05:26:56,907 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:26:56,928 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7669129015d0f95992d76 HTTP/1.1" 200 None 2026-03-30 05:26:56,928 - lava_callback - DEBUG - Processing node 69c7669129015d0f95992d76: baseline-arm64-clabbe 2026-03-30 05:26:56,929 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7669129015d0f95992d76 2026-03-30 05:26:56,929 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:26:56,943 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7669129015d0f95992d76 HTTP/1.1" 200 None 2026-03-30 05:26:56,945 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:26:56,958 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6ae1602375b4506df899e HTTP/1.1" 200 None 2026-03-30 05:26:56,960 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:26:56,981 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7669129015d0f95992d76 HTTP/1.1" 200 None 2026-03-30 05:26:56,982 - lava_callback - ERROR - Error processing callback for node 69c7669129015d0f95992d76: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45376 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48648 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48662 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55918 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55924 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59604 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55582 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41896 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50910 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57160 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57162 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:28:16,997 - lava_callback - INFO - Received callback for node 69c766e729015d0f95992d8a INFO: 172.20.0.1:57166 - "POST /node/69c766e729015d0f95992d8a HTTP/1.0" 202 Accepted 2026-03-30 05:28:17,004 - lava_callback - INFO - Starting async job submit for node 69c766e729015d0f95992d8a 2026-03-30 05:28:17,005 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:28:17,019 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c766e729015d0f95992d8a HTTP/1.1" 200 None 2026-03-30 05:28:17,020 - lava_callback - DEBUG - Processing node 69c766e729015d0f95992d8a: baseline-arm64-clabbe 2026-03-30 05:28:17,020 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c766e729015d0f95992d8a 2026-03-30 05:28:17,021 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:28:17,034 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c766e729015d0f95992d8a HTTP/1.1" 200 None 2026-03-30 05:28:17,036 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:28:17,049 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6ae1602375b4506df899e HTTP/1.1" 200 None 2026-03-30 05:28:17,051 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:28:17,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c766e729015d0f95992d8a HTTP/1.1" 200 None 2026-03-30 05:28:17,074 - lava_callback - ERROR - Error processing callback for node 69c766e729015d0f95992d8a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:42534 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42544 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45278 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60696 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43656 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40730 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40684 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40698 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44186 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44190 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52436 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:29:37,180 - lava_callback - INFO - Received callback for node 69c7673a29015d0f95992d99 INFO: 172.20.0.1:52444 - "POST /node/69c7673a29015d0f95992d99 HTTP/1.0" 202 Accepted 2026-03-30 05:29:37,186 - lava_callback - INFO - Starting async job submit for node 69c7673a29015d0f95992d99 2026-03-30 05:29:37,187 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:29:37,201 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7673a29015d0f95992d99 HTTP/1.1" 200 None 2026-03-30 05:29:37,202 - lava_callback - DEBUG - Processing node 69c7673a29015d0f95992d99: baseline-arm64-clabbe 2026-03-30 05:29:37,202 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7673a29015d0f95992d99 2026-03-30 05:29:37,203 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:29:37,216 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7673a29015d0f95992d99 HTTP/1.1" 200 None 2026-03-30 05:29:37,217 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:29:37,230 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 05:29:37,231 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:29:37,259 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7673a29015d0f95992d99 HTTP/1.1" 200 None 2026-03-30 05:29:37,260 - lava_callback - ERROR - Error processing callback for node 69c7673a29015d0f95992d99: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:58186 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56170 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39420 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39902 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39912 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47616 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47620 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40032 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52302 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57472 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:30:57,122 - lava_callback - INFO - Received callback for node 69c7678c29015d0f95992dae INFO: 172.20.0.1:57488 - "POST /node/69c7678c29015d0f95992dae HTTP/1.0" 202 Accepted 2026-03-30 05:30:57,127 - lava_callback - INFO - Starting async job submit for node 69c7678c29015d0f95992dae 2026-03-30 05:30:57,128 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:30:57,143 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7678c29015d0f95992dae HTTP/1.1" 200 None 2026-03-30 05:30:57,144 - lava_callback - DEBUG - Processing node 69c7678c29015d0f95992dae: baseline-arm64-clabbe 2026-03-30 05:30:57,144 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7678c29015d0f95992dae 2026-03-30 05:30:57,144 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:30:57,158 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7678c29015d0f95992dae HTTP/1.1" 200 None 2026-03-30 05:30:57,159 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:30:57,173 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6ae1602375b4506df899e HTTP/1.1" 200 None 2026-03-30 05:30:57,174 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:30:57,194 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7678c29015d0f95992dae HTTP/1.1" 200 None 2026-03-30 05:30:57,195 - lava_callback - ERROR - Error processing callback for node 69c7678c29015d0f95992dae: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38514 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53070 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53086 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55678 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55688 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43424 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40252 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42774 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58100 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41640 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50752 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50756 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37078 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:32:37,171 - lava_callback - INFO - Received callback for node 69c767e029015d0f95992dba INFO: 172.20.0.1:37090 - "POST /node/69c767e029015d0f95992dba HTTP/1.0" 202 Accepted 2026-03-30 05:32:37,177 - lava_callback - INFO - Starting async job submit for node 69c767e029015d0f95992dba 2026-03-30 05:32:37,177 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:32:37,192 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c767e029015d0f95992dba HTTP/1.1" 200 None 2026-03-30 05:32:37,193 - lava_callback - DEBUG - Processing node 69c767e029015d0f95992dba: baseline-arm64-clabbe 2026-03-30 05:32:37,194 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c767e029015d0f95992dba 2026-03-30 05:32:37,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:32:37,207 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c767e029015d0f95992dba HTTP/1.1" 200 None 2026-03-30 05:32:37,209 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:32:37,223 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6ae1602375b4506df899e HTTP/1.1" 200 None 2026-03-30 05:32:37,224 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:32:37,245 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c767e029015d0f95992dba HTTP/1.1" 200 None 2026-03-30 05:32:37,250 - lava_callback - ERROR - Error processing callback for node 69c767e029015d0f95992dba: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38566 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45242 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48746 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45316 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45328 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38998 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39012 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41720 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41938 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54896 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:33:57,246 - lava_callback - INFO - Received callback for node 69c7683229015d0f95992dc9 INFO: 172.20.0.1:54904 - "POST /node/69c7683229015d0f95992dc9 HTTP/1.0" 202 Accepted 2026-03-30 05:33:57,251 - lava_callback - INFO - Starting async job submit for node 69c7683229015d0f95992dc9 2026-03-30 05:33:57,252 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:33:57,266 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7683229015d0f95992dc9 HTTP/1.1" 200 None 2026-03-30 05:33:57,267 - lava_callback - DEBUG - Processing node 69c7683229015d0f95992dc9: baseline-arm64-clabbe 2026-03-30 05:33:57,268 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7683229015d0f95992dc9 2026-03-30 05:33:57,268 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:33:57,284 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7683229015d0f95992dc9 HTTP/1.1" 200 None 2026-03-30 05:33:57,285 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:33:57,298 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 05:33:57,299 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:33:57,323 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7683229015d0f95992dc9 HTTP/1.1" 200 None 2026-03-30 05:33:57,327 - lava_callback - ERROR - Error processing callback for node 69c7683229015d0f95992dc9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54052 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35176 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35188 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48284 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48294 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39802 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36296 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41124 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48490 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37312 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37328 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57512 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57516 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35126 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51612 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37408 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38292 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34486 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34500 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55628 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55632 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33724 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:36:37,358 - lava_callback - INFO - Received callback for node 69c768d729015d0f95992de9 INFO: 172.20.0.1:33732 - "POST /node/69c768d729015d0f95992de9 HTTP/1.0" 202 Accepted 2026-03-30 05:36:37,364 - lava_callback - INFO - Starting async job submit for node 69c768d729015d0f95992de9 2026-03-30 05:36:37,365 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:36:37,378 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c768d729015d0f95992de9 HTTP/1.1" 200 None 2026-03-30 05:36:37,379 - lava_callback - DEBUG - Processing node 69c768d729015d0f95992de9: baseline-arm64-clabbe 2026-03-30 05:36:37,379 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c768d729015d0f95992de9 2026-03-30 05:36:37,380 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:36:37,393 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c768d729015d0f95992de9 HTTP/1.1" 200 None 2026-03-30 05:36:37,395 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:36:37,422 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 05:36:37,423 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:36:37,445 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c768d729015d0f95992de9 HTTP/1.1" 200 None 2026-03-30 05:36:37,447 - lava_callback - ERROR - Error processing callback for node 69c768d729015d0f95992de9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55392 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59120 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51720 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45488 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45492 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51754 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51760 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60690 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37536 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60318 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55262 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53568 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53572 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41182 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41192 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34992 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49380 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58922 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39960 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55022 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55036 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:39:17,508 - lava_callback - INFO - Received callback for node 69c7697629015d0f95992dff INFO: 172.20.0.1:55044 - "POST /node/69c7697629015d0f95992dff HTTP/1.0" 202 Accepted 2026-03-30 05:39:17,516 - lava_callback - INFO - Starting async job submit for node 69c7697629015d0f95992dff 2026-03-30 05:39:17,517 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:39:17,533 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7697629015d0f95992dff HTTP/1.1" 200 None 2026-03-30 05:39:17,534 - lava_callback - DEBUG - Processing node 69c7697629015d0f95992dff: baseline-arm64-clabbe 2026-03-30 05:39:17,534 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7697629015d0f95992dff 2026-03-30 05:39:17,535 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:39:17,548 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7697629015d0f95992dff HTTP/1.1" 200 None 2026-03-30 05:39:17,549 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:39:17,563 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 05:39:17,565 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:39:17,585 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7697629015d0f95992dff HTTP/1.1" 200 None 2026-03-30 05:39:17,586 - lava_callback - ERROR - Error processing callback for node 69c7697629015d0f95992dff: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:55890 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55892 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53290 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36904 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36880 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48326 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38572 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38576 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54284 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54286 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40440 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:40:38,557 - lava_callback - INFO - Received callback for node 69c769c229015d0f95992e0b INFO: 172.20.0.1:40444 - "POST /node/69c769c229015d0f95992e0b HTTP/1.0" 202 Accepted 2026-03-30 05:40:38,563 - lava_callback - INFO - Starting async job submit for node 69c769c229015d0f95992e0b 2026-03-30 05:40:38,563 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:40:38,577 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c769c229015d0f95992e0b HTTP/1.1" 200 None 2026-03-30 05:40:38,578 - lava_callback - DEBUG - Processing node 69c769c229015d0f95992e0b: baseline-arm64-baylibre 2026-03-30 05:40:38,579 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c769c229015d0f95992e0b 2026-03-30 05:40:38,579 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:40:38,594 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c769c229015d0f95992e0b HTTP/1.1" 200 None 2026-03-30 05:40:38,595 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:40:38,609 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 05:40:38,610 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:40:38,630 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c769c229015d0f95992e0b HTTP/1.1" 200 None 2026-03-30 05:40:38,631 - lava_callback - ERROR - Error processing callback for node 69c769c229015d0f95992e0b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:60770 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36918 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46662 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56432 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56448 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40336 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40342 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45086 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53430 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52824 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:41:57,600 - lava_callback - INFO - Received callback for node 69c76a1929015d0f95992e1b INFO: 172.20.0.1:52828 - "POST /node/69c76a1929015d0f95992e1b HTTP/1.0" 202 Accepted 2026-03-30 05:41:57,606 - lava_callback - INFO - Starting async job submit for node 69c76a1929015d0f95992e1b 2026-03-30 05:41:57,607 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:41:57,621 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76a1929015d0f95992e1b HTTP/1.1" 200 None 2026-03-30 05:41:57,622 - lava_callback - DEBUG - Processing node 69c76a1929015d0f95992e1b: baseline-arm64-clabbe 2026-03-30 05:41:57,622 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c76a1929015d0f95992e1b 2026-03-30 05:41:57,623 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:41:57,636 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76a1929015d0f95992e1b HTTP/1.1" 200 None 2026-03-30 05:41:57,637 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:41:57,650 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 05:41:57,652 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:41:57,672 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c76a1929015d0f95992e1b HTTP/1.1" 200 None 2026-03-30 05:41:57,680 - lava_callback - ERROR - Error processing callback for node 69c76a1929015d0f95992e1b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:50106 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45282 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45292 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53684 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53690 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56676 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37638 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38976 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50918 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59988 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59996 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:43:17,661 - lava_callback - INFO - Received callback for node 69c76a6a29015d0f95992e2a INFO: 172.20.0.1:60002 - "POST /node/69c76a6a29015d0f95992e2a HTTP/1.0" 202 Accepted 2026-03-30 05:43:17,666 - lava_callback - INFO - Starting async job submit for node 69c76a6a29015d0f95992e2a 2026-03-30 05:43:17,667 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:43:17,682 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76a6a29015d0f95992e2a HTTP/1.1" 200 None 2026-03-30 05:43:17,683 - lava_callback - DEBUG - Processing node 69c76a6a29015d0f95992e2a: baseline-arm64-clabbe 2026-03-30 05:43:17,683 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c76a6a29015d0f95992e2a 2026-03-30 05:43:17,684 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:43:17,696 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76a6a29015d0f95992e2a HTTP/1.1" 200 None 2026-03-30 05:43:17,697 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:43:17,710 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 05:43:17,712 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:43:17,734 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c76a6a29015d0f95992e2a HTTP/1.1" 200 None 2026-03-30 05:43:17,735 - lava_callback - ERROR - Error processing callback for node 69c76a6a29015d0f95992e2a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:40098 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40112 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55050 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52416 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41982 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42082 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42094 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54752 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54766 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:44:32,403 - lava_callback - INFO - Received callback for node 69c9ce803c3aafdab4bea22b INFO: 172.20.0.1:60010 - "POST /node/69c9ce803c3aafdab4bea22b HTTP/1.0" 202 Accepted 2026-03-30 05:44:32,409 - lava_callback - INFO - Starting async job submit for node 69c9ce803c3aafdab4bea22b 2026-03-30 05:44:32,426 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:44:32,441 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce803c3aafdab4bea22b HTTP/1.1" 200 None 2026-03-30 05:44:32,442 - lava_callback - DEBUG - Processing node 69c9ce803c3aafdab4bea22b: baseline-arm64 2026-03-30 05:44:32,792 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 05:44:32,971 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 05:44:32,972 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce803c3aafdab4bea22b/log.txt.gz 2026-03-30 05:44:33,175 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 05:44:33,407 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 05:44:33,409 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce803c3aafdab4bea22b/lava-logs.yaml 2026-03-30 05:44:33,417 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 05:44:33,574 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 05:44:33,576 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69c9ce803c3aafdab4bea22b/lava_callback.json.gz 2026-03-30 05:44:33,595 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:44:33,609 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce803c3aafdab4bea22b HTTP/1.1" 200 None 2026-03-30 05:44:33,611 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:44:33,625 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 05:44:33,626 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:44:33,663 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce803c3aafdab4bea22b HTTP/1.1" 200 None 2026-03-30 05:44:33,664 - lava_callback - INFO - Completed processing callback for node 69c9ce803c3aafdab4bea22b INFO: 172.20.0.1:60020 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:44:42,604 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:44:42,633 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:46482 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33466 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54640 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46484 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46492 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34324 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51562 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40642 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57740 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:45:57,733 - lava_callback - INFO - Received callback for node 69c76b0a29015d0f95992e48 INFO: 172.20.0.1:57744 - "POST /node/69c76b0a29015d0f95992e48 HTTP/1.0" 202 Accepted 2026-03-30 05:45:57,739 - lava_callback - INFO - Starting async job submit for node 69c76b0a29015d0f95992e48 2026-03-30 05:45:57,740 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:45:57,754 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76b0a29015d0f95992e48 HTTP/1.1" 200 None 2026-03-30 05:45:57,755 - lava_callback - DEBUG - Processing node 69c76b0a29015d0f95992e48: baseline-arm64-clabbe 2026-03-30 05:45:57,755 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c76b0a29015d0f95992e48 2026-03-30 05:45:57,755 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:45:57,770 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76b0a29015d0f95992e48 HTTP/1.1" 200 None 2026-03-30 05:45:57,771 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:45:57,784 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 05:45:57,786 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:45:57,806 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c76b0a29015d0f95992e48 HTTP/1.1" 200 None 2026-03-30 05:45:57,812 - lava_callback - ERROR - Error processing callback for node 69c76b0a29015d0f95992e48: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45044 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47614 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47626 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47862 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47868 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48686 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34298 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38258 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42974 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56542 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56554 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53756 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53772 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60360 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57974 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50398 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57104 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38514 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38518 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:48:17,808 - lava_callback - INFO - Received callback for node 69c76b9b29015d0f95992e65 INFO: 172.20.0.1:38522 - "POST /node/69c76b9b29015d0f95992e65 HTTP/1.0" 202 Accepted 2026-03-30 05:48:17,814 - lava_callback - INFO - Starting async job submit for node 69c76b9b29015d0f95992e65 2026-03-30 05:48:17,815 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:48:17,829 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76b9b29015d0f95992e65 HTTP/1.1" 200 None 2026-03-30 05:48:17,830 - lava_callback - DEBUG - Processing node 69c76b9b29015d0f95992e65: baseline-arm64-clabbe 2026-03-30 05:48:17,830 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c76b9b29015d0f95992e65 2026-03-30 05:48:17,831 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:48:17,844 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76b9b29015d0f95992e65 HTTP/1.1" 200 None 2026-03-30 05:48:17,845 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:48:17,858 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 05:48:17,859 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:48:17,880 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c76b9b29015d0f95992e65 HTTP/1.1" 200 None 2026-03-30 05:48:17,888 - lava_callback - ERROR - Error processing callback for node 69c76b9b29015d0f95992e65: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46926 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46934 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51800 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:48:46,280 - lava_callback - INFO - Received callback for node 69c9ce903c3aafdab4bea24f INFO: 172.20.0.1:33384 - "POST /node/69c9ce903c3aafdab4bea24f HTTP/1.0" 202 Accepted 2026-03-30 05:48:46,287 - lava_callback - INFO - Starting async job submit for node 69c9ce903c3aafdab4bea24f 2026-03-30 05:48:46,305 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:48:46,320 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce903c3aafdab4bea24f HTTP/1.1" 200 None 2026-03-30 05:48:46,320 - lava_callback - DEBUG - Processing node 69c9ce903c3aafdab4bea24f: baseline-nfs-arm64 INFO: 172.20.0.1:33396 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:48:46,713 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 05:48:46,920 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 05:48:46,922 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce903c3aafdab4bea24f/log.txt.gz 2026-03-30 05:48:47,157 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 05:48:47,401 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 05:48:47,403 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce903c3aafdab4bea24f/lava-logs.yaml 2026-03-30 05:48:47,413 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-30 05:48:47,598 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-30 05:48:47,599 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69c9ce903c3aafdab4bea24f/lava_callback.json.gz 2026-03-30 05:48:47,619 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:48:47,634 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9ce903c3aafdab4bea24f HTTP/1.1" 200 None 2026-03-30 05:48:47,635 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:48:47,649 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c9c1283c3aafdab4be7756 HTTP/1.1" 200 None 2026-03-30 05:48:47,651 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:48:47,681 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c9ce903c3aafdab4bea24f HTTP/1.1" 200 None 2026-03-30 05:48:47,683 - lava_callback - INFO - Completed processing callback for node 69c9ce903c3aafdab4bea24f INFO: 172.20.0.1:40914 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40476 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:49:12,636 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:49:12,652 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.20.0.1:38308 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38322 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54186 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54198 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57726 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:49:37,933 - lava_callback - INFO - Received callback for node 69c76bed29015d0f95992e72 INFO: 172.20.0.1:57734 - "POST /node/69c76bed29015d0f95992e72 HTTP/1.0" 202 Accepted 2026-03-30 05:49:37,939 - lava_callback - INFO - Starting async job submit for node 69c76bed29015d0f95992e72 2026-03-30 05:49:37,940 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:49:37,963 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76bed29015d0f95992e72 HTTP/1.1" 200 None 2026-03-30 05:49:37,964 - lava_callback - DEBUG - Processing node 69c76bed29015d0f95992e72: baseline-arm64-clabbe 2026-03-30 05:49:37,964 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c76bed29015d0f95992e72 2026-03-30 05:49:37,965 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:49:37,977 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76bed29015d0f95992e72 HTTP/1.1" 200 None 2026-03-30 05:49:37,979 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:49:38,003 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 05:49:38,005 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:49:38,025 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c76bed29015d0f95992e72 HTTP/1.1" 200 None 2026-03-30 05:49:38,026 - lava_callback - ERROR - Error processing callback for node 69c76bed29015d0f95992e72: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:51454 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59456 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59970 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37484 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37492 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44152 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44154 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34870 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40284 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36574 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56926 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39658 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39674 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58048 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58056 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46992 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35272 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45320 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:51:59,206 - lava_callback - INFO - Received callback for node 69c76c7329015d0f95992e97 INFO: 172.20.0.1:45322 - "POST /node/69c76c7329015d0f95992e97 HTTP/1.0" 202 Accepted 2026-03-30 05:51:59,211 - lava_callback - INFO - Starting async job submit for node 69c76c7329015d0f95992e97 2026-03-30 05:51:59,212 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:51:59,227 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76c7329015d0f95992e97 HTTP/1.1" 200 None 2026-03-30 05:51:59,228 - lava_callback - DEBUG - Processing node 69c76c7329015d0f95992e97: baseline-arm-baylibre 2026-03-30 05:51:59,229 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c76c7329015d0f95992e97 2026-03-30 05:51:59,229 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:51:59,249 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76c7329015d0f95992e97 HTTP/1.1" 200 None 2026-03-30 05:51:59,250 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:51:59,264 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6ae0102375b4506df898a HTTP/1.1" 200 None 2026-03-30 05:51:59,266 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:51:59,287 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c76c7329015d0f95992e97 HTTP/1.1" 200 None 2026-03-30 05:51:59,293 - lava_callback - ERROR - Error processing callback for node 69c76c7329015d0f95992e97: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:39126 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53884 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53900 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44972 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44982 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49960 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:52:39,104 - lava_callback - INFO - Received callback for node 69c76c9929015d0f95992ea4 INFO: 172.20.0.1:49976 - "POST /node/69c76c9929015d0f95992ea4 HTTP/1.0" 202 Accepted 2026-03-30 05:52:39,110 - lava_callback - INFO - Starting async job submit for node 69c76c9929015d0f95992ea4 2026-03-30 05:52:39,111 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:52:39,125 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76c9929015d0f95992ea4 HTTP/1.1" 200 None 2026-03-30 05:52:39,126 - lava_callback - DEBUG - Processing node 69c76c9929015d0f95992ea4: baseline-arm-baylibre 2026-03-30 05:52:39,126 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c76c9929015d0f95992ea4 2026-03-30 05:52:39,127 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:52:39,148 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76c9929015d0f95992ea4 HTTP/1.1" 200 None 2026-03-30 05:52:39,149 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:52:39,162 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6ae0102375b4506df898a HTTP/1.1" 200 None 2026-03-30 05:52:39,163 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:52:39,183 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c76c9929015d0f95992ea4 HTTP/1.1" 200 None 2026-03-30 05:52:39,193 - lava_callback - ERROR - Error processing callback for node 69c76c9929015d0f95992ea4: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48662 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41304 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41812 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57636 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57644 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47482 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47484 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57368 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41362 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58596 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53450 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48030 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48036 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39942 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39956 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56584 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49740 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56618 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46646 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46622 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46626 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59852 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59868 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43962 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:55:38,188 - lava_callback - INFO - Received callback for node 69c76d5029015d0f95992edd INFO: 172.20.0.1:43972 - "POST /node/69c76d5029015d0f95992edd HTTP/1.0" 202 Accepted 2026-03-30 05:55:38,194 - lava_callback - INFO - Starting async job submit for node 69c76d5029015d0f95992edd 2026-03-30 05:55:38,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:55:38,209 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76d5029015d0f95992edd HTTP/1.1" 200 None 2026-03-30 05:55:38,210 - lava_callback - DEBUG - Processing node 69c76d5029015d0f95992edd: baseline-arm-clabbe 2026-03-30 05:55:38,210 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c76d5029015d0f95992edd 2026-03-30 05:55:38,211 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:55:38,223 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76d5029015d0f95992edd HTTP/1.1" 200 None 2026-03-30 05:55:38,225 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:55:38,238 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6ae0102375b4506df898a HTTP/1.1" 200 None 2026-03-30 05:55:38,239 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:55:38,268 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c76d5029015d0f95992edd HTTP/1.1" 200 None 2026-03-30 05:55:38,269 - lava_callback - ERROR - Error processing callback for node 69c76d5029015d0f95992edd: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:50492 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60504 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46970 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46980 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:56:18,155 - lava_callback - INFO - Received callback for node 69c76d7529015d0f95992eea INFO: 172.20.0.1:46988 - "POST /node/69c76d7529015d0f95992eea HTTP/1.0" 202 Accepted 2026-03-30 05:56:18,161 - lava_callback - INFO - Starting async job submit for node 69c76d7529015d0f95992eea 2026-03-30 05:56:18,162 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:56:18,177 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76d7529015d0f95992eea HTTP/1.1" 200 None 2026-03-30 05:56:18,178 - lava_callback - DEBUG - Processing node 69c76d7529015d0f95992eea: baseline-arm-clabbe 2026-03-30 05:56:18,178 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c76d7529015d0f95992eea 2026-03-30 05:56:18,179 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:56:18,193 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76d7529015d0f95992eea HTTP/1.1" 200 None 2026-03-30 05:56:18,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:56:18,208 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6ae0102375b4506df898a HTTP/1.1" 200 None 2026-03-30 05:56:18,209 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:56:18,236 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c76d7529015d0f95992eea HTTP/1.1" 200 None 2026-03-30 05:56:18,238 - lava_callback - ERROR - Error processing callback for node 69c76d7529015d0f95992eea: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:51354 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51362 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40442 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55552 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46476 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:56:58,221 - lava_callback - INFO - Received callback for node 69c76d9a29015d0f95992ef7 INFO: 172.20.0.1:46492 - "POST /node/69c76d9a29015d0f95992ef7 HTTP/1.0" 202 Accepted 2026-03-30 05:56:58,227 - lava_callback - INFO - Starting async job submit for node 69c76d9a29015d0f95992ef7 2026-03-30 05:56:58,228 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:56:58,242 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76d9a29015d0f95992ef7 HTTP/1.1" 200 None 2026-03-30 05:56:58,243 - lava_callback - DEBUG - Processing node 69c76d9a29015d0f95992ef7: baseline-arm-clabbe 2026-03-30 05:56:58,243 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c76d9a29015d0f95992ef7 2026-03-30 05:56:58,243 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:56:58,259 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76d9a29015d0f95992ef7 HTTP/1.1" 200 None 2026-03-30 05:56:58,261 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:56:58,276 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6ae0102375b4506df898a HTTP/1.1" 200 None 2026-03-30 05:56:58,278 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:56:58,309 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c76d9a29015d0f95992ef7 HTTP/1.1" 200 None 2026-03-30 05:56:58,310 - lava_callback - ERROR - Error processing callback for node 69c76d9a29015d0f95992ef7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41486 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39452 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39468 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51362 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51364 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43346 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:57:38,227 - lava_callback - INFO - Received callback for node 69c76dc029015d0f95992f0d INFO: 172.20.0.1:43360 - "POST /node/69c76dc029015d0f95992f0d HTTP/1.0" 202 Accepted 2026-03-30 05:57:38,232 - lava_callback - INFO - Starting async job submit for node 69c76dc029015d0f95992f0d 2026-03-30 05:57:38,233 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:57:38,247 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76dc029015d0f95992f0d HTTP/1.1" 200 None 2026-03-30 05:57:38,248 - lava_callback - DEBUG - Processing node 69c76dc029015d0f95992f0d: baseline-arm-clabbe 2026-03-30 05:57:38,248 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c76dc029015d0f95992f0d 2026-03-30 05:57:38,249 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:57:38,283 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76dc029015d0f95992f0d HTTP/1.1" 200 None 2026-03-30 05:57:38,284 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:57:38,297 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6ae0102375b4506df898a HTTP/1.1" 200 None 2026-03-30 05:57:38,299 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:57:38,332 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c76dc029015d0f95992f0d HTTP/1.1" 200 None 2026-03-30 05:57:38,337 - lava_callback - ERROR - Error processing callback for node 69c76dc029015d0f95992f0d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47122 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45620 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52728 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52136 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52152 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:58:18,307 - lava_callback - INFO - Received callback for node 69c76de629015d0f95992f16 INFO: 172.20.0.1:52166 - "POST /node/69c76de629015d0f95992f16 HTTP/1.0" 202 Accepted 2026-03-30 05:58:18,312 - lava_callback - INFO - Starting async job submit for node 69c76de629015d0f95992f16 2026-03-30 05:58:18,313 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:58:18,330 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76de629015d0f95992f16 HTTP/1.1" 200 None 2026-03-30 05:58:18,331 - lava_callback - DEBUG - Processing node 69c76de629015d0f95992f16: baseline-arm-clabbe 2026-03-30 05:58:18,331 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c76de629015d0f95992f16 2026-03-30 05:58:18,332 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:58:18,346 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76de629015d0f95992f16 HTTP/1.1" 200 None 2026-03-30 05:58:18,347 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:58:18,360 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c6ae0102375b4506df898a HTTP/1.1" 200 None 2026-03-30 05:58:18,362 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:58:18,389 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c76de629015d0f95992f16 HTTP/1.1" 200 None 2026-03-30 05:58:18,402 - lava_callback - ERROR - Error processing callback for node 69c76de629015d0f95992f16: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:53550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53554 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55072 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38680 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40818 - "GET / HTTP/1.0" 200 OK 2026-03-30 05:58:58,269 - lava_callback - INFO - Received callback for node 69c76e1929015d0f95992f25 INFO: 172.20.0.1:40824 - "POST /node/69c76e1929015d0f95992f25 HTTP/1.0" 202 Accepted 2026-03-30 05:58:58,275 - lava_callback - INFO - Starting async job submit for node 69c76e1929015d0f95992f25 2026-03-30 05:58:58,276 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:58:58,290 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76e1929015d0f95992f25 HTTP/1.1" 200 None 2026-03-30 05:58:58,291 - lava_callback - DEBUG - Processing node 69c76e1929015d0f95992f25: baseline-arm64-clabbe 2026-03-30 05:58:58,291 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c76e1929015d0f95992f25 2026-03-30 05:58:58,291 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:58:58,320 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76e1929015d0f95992f25 HTTP/1.1" 200 None 2026-03-30 05:58:58,322 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:58:58,335 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 05:58:58,336 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 05:58:58,357 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c76e1929015d0f95992f25 HTTP/1.1" 200 None 2026-03-30 05:58:58,362 - lava_callback - ERROR - Error processing callback for node 69c76e1929015d0f95992f25: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41842 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52398 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52404 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59136 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59144 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37282 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37802 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50386 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60344 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52014 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52024 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:00:18,479 - lava_callback - INFO - Received callback for node 69c76e6a29015d0f95992f34 INFO: 172.20.0.1:52028 - "POST /node/69c76e6a29015d0f95992f34 HTTP/1.0" 202 Accepted 2026-03-30 06:00:18,485 - lava_callback - INFO - Starting async job submit for node 69c76e6a29015d0f95992f34 2026-03-30 06:00:18,485 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:00:18,500 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76e6a29015d0f95992f34 HTTP/1.1" 200 None 2026-03-30 06:00:18,501 - lava_callback - DEBUG - Processing node 69c76e6a29015d0f95992f34: baseline-arm64-clabbe 2026-03-30 06:00:18,501 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c76e6a29015d0f95992f34 2026-03-30 06:00:18,502 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:00:18,515 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76e6a29015d0f95992f34 HTTP/1.1" 200 None 2026-03-30 06:00:18,517 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:00:18,529 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 06:00:18,531 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:00:18,551 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c76e6a29015d0f95992f34 HTTP/1.1" 200 None 2026-03-30 06:00:18,556 - lava_callback - ERROR - Error processing callback for node 69c76e6a29015d0f95992f34: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54746 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54748 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60618 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51772 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41532 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44380 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36978 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36986 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53030 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53046 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39916 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54352 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39244 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:01:58,419 - lava_callback - INFO - Received callback for node 69c76ec129015d0f95992f41 INFO: 172.20.0.1:39260 - "POST /node/69c76ec129015d0f95992f41 HTTP/1.0" 202 Accepted 2026-03-30 06:01:58,424 - lava_callback - INFO - Starting async job submit for node 69c76ec129015d0f95992f41 2026-03-30 06:01:58,425 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:01:58,440 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76ec129015d0f95992f41 HTTP/1.1" 200 None 2026-03-30 06:01:58,440 - lava_callback - DEBUG - Processing node 69c76ec129015d0f95992f41: baseline-arm64-clabbe 2026-03-30 06:01:58,441 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c76ec129015d0f95992f41 2026-03-30 06:01:58,441 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:01:58,455 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76ec129015d0f95992f41 HTTP/1.1" 200 None 2026-03-30 06:01:58,456 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:01:58,475 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 06:01:58,476 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:01:58,501 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c76ec129015d0f95992f41 HTTP/1.1" 200 None 2026-03-30 06:01:58,506 - lava_callback - ERROR - Error processing callback for node 69c76ec129015d0f95992f41: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44474 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41762 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41764 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44400 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44402 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38492 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54602 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52176 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55098 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49892 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49894 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42572 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42586 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49020 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47222 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46598 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38076 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51634 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51648 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51132 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51144 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39624 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47080 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47172 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56628 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56642 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46448 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46462 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59148 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34504 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51318 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44486 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45422 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45426 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56232 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56234 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34040 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45894 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45312 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:06:59,948 - lava_callback - INFO - Received callback for node 69c76ff829015d0f95992f6c INFO: 172.20.0.1:49586 - "POST /node/69c76ff829015d0f95992f6c HTTP/1.0" 202 Accepted 2026-03-30 06:06:59,953 - lava_callback - INFO - Starting async job submit for node 69c76ff829015d0f95992f6c 2026-03-30 06:06:59,954 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:06:59,969 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76ff829015d0f95992f6c HTTP/1.1" 200 None 2026-03-30 06:06:59,970 - lava_callback - DEBUG - Processing node 69c76ff829015d0f95992f6c: baseline-arm64-baylibre 2026-03-30 06:06:59,970 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c76ff829015d0f95992f6c 2026-03-30 06:06:59,970 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:06:59,984 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c76ff829015d0f95992f6c HTTP/1.1" 200 None 2026-03-30 06:06:59,985 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:06:59,998 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 06:07:00,000 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:07:00,026 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c76ff829015d0f95992f6c HTTP/1.1" 200 None 2026-03-30 06:07:00,031 - lava_callback - ERROR - Error processing callback for node 69c76ff829015d0f95992f6c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49596 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57132 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57136 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37646 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37654 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37460 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58074 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49908 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51708 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60692 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60702 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54702 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54710 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49796 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:08:38,700 - lava_callback - INFO - Received callback for node 69c7705129015d0f95992f7b INFO: 172.20.0.1:49812 - "POST /node/69c7705129015d0f95992f7b HTTP/1.0" 202 Accepted 2026-03-30 06:08:38,706 - lava_callback - INFO - Starting async job submit for node 69c7705129015d0f95992f7b 2026-03-30 06:08:38,706 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:08:38,721 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7705129015d0f95992f7b HTTP/1.1" 200 None 2026-03-30 06:08:38,722 - lava_callback - DEBUG - Processing node 69c7705129015d0f95992f7b: baseline-arm64-clabbe 2026-03-30 06:08:38,722 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7705129015d0f95992f7b 2026-03-30 06:08:38,722 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:08:38,747 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7705129015d0f95992f7b HTTP/1.1" 200 None 2026-03-30 06:08:38,749 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:08:38,792 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 06:08:38,794 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:08:38,837 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7705129015d0f95992f7b HTTP/1.1" 200 None 2026-03-30 06:08:38,842 - lava_callback - ERROR - Error processing callback for node 69c7705129015d0f95992f7b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41158 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34968 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37466 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49040 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49046 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57794 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57804 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41278 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45762 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34778 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:09:58,737 - lava_callback - INFO - Received callback for node 69c770a429015d0f95992f8b INFO: 172.20.0.1:34792 - "POST /node/69c770a429015d0f95992f8b HTTP/1.0" 202 Accepted 2026-03-30 06:09:58,744 - lava_callback - INFO - Starting async job submit for node 69c770a429015d0f95992f8b 2026-03-30 06:09:58,745 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:09:58,785 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c770a429015d0f95992f8b HTTP/1.1" 200 None 2026-03-30 06:09:58,786 - lava_callback - DEBUG - Processing node 69c770a429015d0f95992f8b: baseline-arm64-clabbe 2026-03-30 06:09:58,786 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c770a429015d0f95992f8b 2026-03-30 06:09:58,787 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:09:58,801 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c770a429015d0f95992f8b HTTP/1.1" 200 None 2026-03-30 06:09:58,802 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:09:58,815 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 06:09:58,816 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:09:58,841 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c770a429015d0f95992f8b HTTP/1.1" 200 None 2026-03-30 06:09:58,843 - lava_callback - ERROR - Error processing callback for node 69c770a429015d0f95992f8b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48540 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56324 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:56328 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33740 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33748 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32802 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39992 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46370 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55878 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46124 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46128 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:11:18,758 - lava_callback - INFO - Received callback for node 69c770f729015d0f95992f9c INFO: 172.20.0.1:46132 - "POST /node/69c770f729015d0f95992f9c HTTP/1.0" 202 Accepted 2026-03-30 06:11:18,763 - lava_callback - INFO - Starting async job submit for node 69c770f729015d0f95992f9c 2026-03-30 06:11:18,764 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:11:18,828 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c770f729015d0f95992f9c HTTP/1.1" 200 None 2026-03-30 06:11:18,829 - lava_callback - DEBUG - Processing node 69c770f729015d0f95992f9c: baseline-arm64-clabbe 2026-03-30 06:11:18,829 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c770f729015d0f95992f9c 2026-03-30 06:11:18,830 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:11:18,844 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c770f729015d0f95992f9c HTTP/1.1" 200 None 2026-03-30 06:11:18,845 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:11:18,871 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 06:11:18,872 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:11:18,899 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c770f729015d0f95992f9c HTTP/1.1" 200 None 2026-03-30 06:11:18,901 - lava_callback - ERROR - Error processing callback for node 69c770f729015d0f95992f9c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48222 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48236 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50194 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37186 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57160 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38980 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47478 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47488 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42246 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42262 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36390 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:12:38,883 - lava_callback - INFO - Received callback for node 69c7714929015d0f95992fab INFO: 172.20.0.1:36398 - "POST /node/69c7714929015d0f95992fab HTTP/1.0" 202 Accepted 2026-03-30 06:12:38,889 - lava_callback - INFO - Starting async job submit for node 69c7714929015d0f95992fab 2026-03-30 06:12:38,890 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:12:38,906 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7714929015d0f95992fab HTTP/1.1" 200 None 2026-03-30 06:12:38,906 - lava_callback - DEBUG - Processing node 69c7714929015d0f95992fab: baseline-arm64-clabbe 2026-03-30 06:12:38,907 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7714929015d0f95992fab 2026-03-30 06:12:38,907 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:12:38,921 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7714929015d0f95992fab HTTP/1.1" 200 None 2026-03-30 06:12:38,922 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:12:38,996 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 06:12:38,998 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:12:39,020 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7714929015d0f95992fab HTTP/1.1" 200 None 2026-03-30 06:12:39,021 - lava_callback - ERROR - Error processing callback for node 69c7714929015d0f95992fab: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41688 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59588 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54302 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38414 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38424 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53820 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53826 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56580 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41128 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60804 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:13:58,942 - lava_callback - INFO - Received callback for node 69c7719429015d0f95992fb1 INFO: 172.20.0.1:60806 - "POST /node/69c7719429015d0f95992fb1 HTTP/1.0" 202 Accepted 2026-03-30 06:13:58,948 - lava_callback - INFO - Starting async job submit for node 69c7719429015d0f95992fb1 2026-03-30 06:13:58,948 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:13:58,963 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7719429015d0f95992fb1 HTTP/1.1" 200 None 2026-03-30 06:13:58,964 - lava_callback - DEBUG - Processing node 69c7719429015d0f95992fb1: baseline-arm64-clabbe 2026-03-30 06:13:58,965 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7719429015d0f95992fb1 2026-03-30 06:13:58,965 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:13:58,983 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7719429015d0f95992fb1 HTTP/1.1" 200 None 2026-03-30 06:13:58,984 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:13:58,998 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 06:13:58,999 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:13:59,020 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7719429015d0f95992fb1 HTTP/1.1" 200 None 2026-03-30 06:13:59,032 - lava_callback - ERROR - Error processing callback for node 69c7719429015d0f95992fb1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:56974 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42840 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42846 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33920 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33926 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39792 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55576 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51926 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41474 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55776 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55778 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:15:18,979 - lava_callback - INFO - Received callback for node 69c771e529015d0f95992fbe INFO: 172.20.0.1:55782 - "POST /node/69c771e529015d0f95992fbe HTTP/1.0" 202 Accepted 2026-03-30 06:15:18,985 - lava_callback - INFO - Starting async job submit for node 69c771e529015d0f95992fbe 2026-03-30 06:15:18,986 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:15:19,000 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c771e529015d0f95992fbe HTTP/1.1" 200 None 2026-03-30 06:15:19,001 - lava_callback - DEBUG - Processing node 69c771e529015d0f95992fbe: baseline-arm64-clabbe 2026-03-30 06:15:19,001 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c771e529015d0f95992fbe 2026-03-30 06:15:19,002 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:15:19,015 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c771e529015d0f95992fbe HTTP/1.1" 200 None 2026-03-30 06:15:19,016 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:15:19,039 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c40ad9944faaf0e7f5e41d HTTP/1.1" 200 None 2026-03-30 06:15:19,041 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:15:19,083 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c771e529015d0f95992fbe HTTP/1.1" 200 None 2026-03-30 06:15:19,091 - lava_callback - ERROR - Error processing callback for node 69c771e529015d0f95992fbe: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38062 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38064 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60568 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37658 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37426 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59878 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44452 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44454 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36954 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36968 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58542 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46874 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33932 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45816 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53628 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53638 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40160 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40164 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37276 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41906 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53442 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35844 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40208 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40216 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54712 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54722 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43870 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51528 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51908 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42908 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35560 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35576 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33546 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33562 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59062 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53628 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48556 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56282 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51702 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51706 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60912 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60916 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50008 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37930 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46958 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34502 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45486 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45494 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57640 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57646 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51576 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57494 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60932 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38514 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34638 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34640 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58306 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58312 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40592 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53644 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38120 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52368 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36990 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36996 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35994 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36010 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49252 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35682 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43974 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35280 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59112 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59122 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49344 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49348 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55106 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55850 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36008 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49294 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49070 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49086 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51158 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51162 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52154 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45790 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34950 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41038 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33252 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33264 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34144 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34148 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56862 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36312 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48458 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49790 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49452 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49460 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58068 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58076 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55960 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48786 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54226 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59800 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55796 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55804 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51844 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51846 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50238 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55912 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46462 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36628 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37310 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37324 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49326 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36702 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47416 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45216 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58034 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44892 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44900 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54858 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54874 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35160 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34720 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32778 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60074 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49506 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49510 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34562 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34578 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55430 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48016 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54740 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34294 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54398 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54414 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47458 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47466 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36160 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55880 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50042 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44446 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44458 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33062 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33076 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38822 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53830 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40158 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49214 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38566 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38572 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53110 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53114 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51568 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39476 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44940 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49508 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59144 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59148 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37800 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37804 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48052 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33254 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51462 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53460 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36844 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36856 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36862 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41138 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41150 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37362 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53684 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37310 - "GET /favicon.ico HTTP/1.0" 404 Not Found INFO: 172.20.0.1:37314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45476 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60728 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60738 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48488 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48502 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39226 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33454 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39794 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51866 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47952 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47966 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48820 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48826 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51760 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51774 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51788 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50150 - "GET /favicon.ico HTTP/1.0" 404 Not Found INFO: 172.20.0.1:50158 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56498 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33716 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44528 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44532 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45230 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45232 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53592 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36562 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43164 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:40:01,764 - lava_callback - INFO - Received callback for node 69c777b929015d0f9599302b INFO: 172.20.0.1:33686 - "POST /node/69c777b929015d0f9599302b HTTP/1.0" 202 Accepted 2026-03-30 06:40:01,769 - lava_callback - INFO - Starting async job submit for node 69c777b929015d0f9599302b 2026-03-30 06:40:01,770 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:40:01,783 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c777b929015d0f9599302b HTTP/1.1" 200 None 2026-03-30 06:40:01,784 - lava_callback - DEBUG - Processing node 69c777b929015d0f9599302b: baseline-arm-baylibre 2026-03-30 06:40:01,784 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c777b929015d0f9599302b 2026-03-30 06:40:01,785 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:40:01,798 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c777b929015d0f9599302b HTTP/1.1" 200 None 2026-03-30 06:40:01,800 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:40:01,994 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 06:40:01,996 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:40:02,038 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c777b929015d0f9599302b HTTP/1.1" 200 None 2026-03-30 06:40:02,040 - lava_callback - ERROR - Error processing callback for node 69c777b929015d0f9599302b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:33688 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47722 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47730 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37544 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37548 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35342 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44438 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50102 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:41:01,720 - lava_callback - INFO - Received callback for node 69c777f029015d0f9599303b INFO: 172.20.0.1:35934 - "POST /node/69c777f029015d0f9599303b HTTP/1.0" 202 Accepted 2026-03-30 06:41:01,725 - lava_callback - INFO - Starting async job submit for node 69c777f029015d0f9599303b 2026-03-30 06:41:01,726 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:41:01,741 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c777f029015d0f9599303b HTTP/1.1" 200 None 2026-03-30 06:41:01,741 - lava_callback - DEBUG - Processing node 69c777f029015d0f9599303b: baseline-arm-baylibre 2026-03-30 06:41:01,742 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c777f029015d0f9599303b 2026-03-30 06:41:01,742 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:41:01,755 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c777f029015d0f9599303b HTTP/1.1" 200 None 2026-03-30 06:41:01,756 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:41:01,778 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 06:41:01,779 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:41:01,799 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c777f029015d0f9599303b HTTP/1.1" 200 None 2026-03-30 06:41:01,805 - lava_callback - ERROR - Error processing callback for node 69c777f029015d0f9599303b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35946 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39572 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39578 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44110 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44126 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39834 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:41:41,828 - lava_callback - INFO - Received callback for node 69c7781629015d0f95993049 INFO: 172.20.0.1:45692 - "POST /node/69c7781629015d0f95993049 HTTP/1.0" 202 Accepted 2026-03-30 06:41:41,833 - lava_callback - INFO - Starting async job submit for node 69c7781629015d0f95993049 2026-03-30 06:41:41,834 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:41:41,856 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7781629015d0f95993049 HTTP/1.1" 200 None 2026-03-30 06:41:41,858 - lava_callback - DEBUG - Processing node 69c7781629015d0f95993049: baseline-arm-baylibre 2026-03-30 06:41:41,858 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7781629015d0f95993049 2026-03-30 06:41:41,859 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:41:41,872 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7781629015d0f95993049 HTTP/1.1" 200 None 2026-03-30 06:41:41,873 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:41:41,886 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 06:41:41,887 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:41:41,907 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7781629015d0f95993049 HTTP/1.1" 200 None 2026-03-30 06:41:41,909 - lava_callback - ERROR - Error processing callback for node 69c7781629015d0f95993049: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45700 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47492 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41686 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54220 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54224 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:42:21,812 - lava_callback - INFO - Received callback for node 69c7783b29015d0f9599305a INFO: 172.20.0.1:36946 - "POST /node/69c7783b29015d0f9599305a HTTP/1.0" 202 Accepted 2026-03-30 06:42:21,817 - lava_callback - INFO - Starting async job submit for node 69c7783b29015d0f9599305a 2026-03-30 06:42:21,818 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:42:21,833 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7783b29015d0f9599305a HTTP/1.1" 200 None 2026-03-30 06:42:21,834 - lava_callback - DEBUG - Processing node 69c7783b29015d0f9599305a: baseline-arm-baylibre 2026-03-30 06:42:21,834 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7783b29015d0f9599305a 2026-03-30 06:42:21,835 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:42:21,849 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7783b29015d0f9599305a HTTP/1.1" 200 None 2026-03-30 06:42:21,850 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:42:21,864 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 06:42:21,865 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:42:21,885 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7783b29015d0f9599305a HTTP/1.1" 200 None 2026-03-30 06:42:21,886 - lava_callback - ERROR - Error processing callback for node 69c7783b29015d0f9599305a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:36948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36950 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41302 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:42:41,887 - lava_callback - INFO - Received callback for node 69c7786029015d0f95993068 INFO: 172.20.0.1:51708 - "POST /node/69c7786029015d0f95993068 HTTP/1.0" 202 Accepted 2026-03-30 06:42:41,893 - lava_callback - INFO - Starting async job submit for node 69c7786029015d0f95993068 2026-03-30 06:42:41,894 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:42:41,909 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7786029015d0f95993068 HTTP/1.1" 200 None 2026-03-30 06:42:41,910 - lava_callback - DEBUG - Processing node 69c7786029015d0f95993068: baseline-arm-baylibre 2026-03-30 06:42:41,911 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7786029015d0f95993068 2026-03-30 06:42:41,911 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:42:41,927 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7786029015d0f95993068 HTTP/1.1" 200 None 2026-03-30 06:42:41,928 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:42:41,943 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 06:42:41,944 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:42:41,977 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7786029015d0f95993068 HTTP/1.1" 200 None 2026-03-30 06:42:41,979 - lava_callback - ERROR - Error processing callback for node 69c7786029015d0f95993068: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:51722 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43052 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53884 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34624 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34632 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50164 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50170 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43748 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43760 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39288 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55006 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50750 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44778 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44792 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39312 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39314 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41368 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58640 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43816 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56146 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53896 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53912 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39364 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39366 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35792 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55960 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54416 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58646 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55368 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55380 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60292 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60300 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51346 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:46:42,064 - lava_callback - INFO - Received callback for node 69c7794629015d0f95993091 INFO: 172.20.0.1:35450 - "POST /node/69c7794629015d0f95993091 HTTP/1.0" 202 Accepted 2026-03-30 06:46:42,070 - lava_callback - INFO - Starting async job submit for node 69c7794629015d0f95993091 2026-03-30 06:46:42,071 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:46:42,085 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7794629015d0f95993091 HTTP/1.1" 200 None 2026-03-30 06:46:42,086 - lava_callback - DEBUG - Processing node 69c7794629015d0f95993091: baseline-arm-baylibre 2026-03-30 06:46:42,086 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7794629015d0f95993091 2026-03-30 06:46:42,086 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:46:42,099 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7794629015d0f95993091 HTTP/1.1" 200 None 2026-03-30 06:46:42,101 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:46:42,114 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 06:46:42,115 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:46:42,136 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7794629015d0f95993091 HTTP/1.1" 200 None 2026-03-30 06:46:42,137 - lava_callback - ERROR - Error processing callback for node 69c7794629015d0f95993091: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40486 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59682 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48488 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48502 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52448 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47878 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46824 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42592 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52012 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35168 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35178 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40246 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40260 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59754 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33368 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35902 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36990 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53022 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53034 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46370 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46374 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50012 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44680 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38894 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58330 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49406 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49416 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46222 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46230 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48706 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56010 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38324 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49432 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49440 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:51:22,317 - lava_callback - INFO - Received callback for node 69c77a6629015d0f9599310f INFO: 172.20.0.1:45552 - "POST /node/69c77a6629015d0f9599310f HTTP/1.0" 202 Accepted 2026-03-30 06:51:22,322 - lava_callback - INFO - Starting async job submit for node 69c77a6629015d0f9599310f 2026-03-30 06:51:22,323 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:51:22,362 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77a6629015d0f9599310f HTTP/1.1" 200 None 2026-03-30 06:51:22,363 - lava_callback - DEBUG - Processing node 69c77a6629015d0f9599310f: baseline-arm-baylibre 2026-03-30 06:51:22,364 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77a6629015d0f9599310f 2026-03-30 06:51:22,364 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:51:22,386 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77a6629015d0f9599310f HTTP/1.1" 200 None 2026-03-30 06:51:22,388 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:51:22,417 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e3a29015d0f9598f73c HTTP/1.1" 200 None 2026-03-30 06:51:22,419 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:51:22,441 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77a6629015d0f9599310f HTTP/1.1" 200 None 2026-03-30 06:51:22,443 - lava_callback - ERROR - Error processing callback for node 69c77a6629015d0f9599310f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45564 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45568 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56152 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59838 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59944 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:52:02,652 - lava_callback - INFO - Received callback for node 69c77a8a29015d0f95993121 INFO: 172.20.0.1:59672 - "POST /node/69c77a8a29015d0f95993121 HTTP/1.0" 202 Accepted 2026-03-30 06:52:02,657 - lava_callback - INFO - Starting async job submit for node 69c77a8a29015d0f95993121 2026-03-30 06:52:02,658 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:52:02,672 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77a8a29015d0f95993121 HTTP/1.1" 200 None 2026-03-30 06:52:02,673 - lava_callback - DEBUG - Processing node 69c77a8a29015d0f95993121: baseline-arm-baylibre 2026-03-30 06:52:02,673 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77a8a29015d0f95993121 2026-03-30 06:52:02,674 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:52:02,687 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77a8a29015d0f95993121 HTTP/1.1" 200 None 2026-03-30 06:52:02,689 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:52:02,713 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e3a29015d0f9598f73c HTTP/1.1" 200 None 2026-03-30 06:52:02,715 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:52:02,736 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77a8a29015d0f95993121 HTTP/1.1" 200 None 2026-03-30 06:52:02,742 - lava_callback - ERROR - Error processing callback for node 69c77a8a29015d0f95993121: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59682 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37370 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37386 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41732 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41738 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52666 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48326 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49456 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36690 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33360 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33372 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38648 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38662 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37218 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60926 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48474 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37356 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48318 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48322 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38554 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38564 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40022 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44610 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36212 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57894 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57900 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39888 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39898 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53678 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:55:40,656 - lava_callback - INFO - Received callback for node 69c77b6429015d0f95993182 INFO: 172.20.0.1:48264 - "POST /node/69c77b6429015d0f95993182 HTTP/1.0" 202 Accepted 2026-03-30 06:55:40,662 - lava_callback - INFO - Starting async job submit for node 69c77b6429015d0f95993182 2026-03-30 06:55:40,663 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:55:40,678 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77b6429015d0f95993182 HTTP/1.1" 200 None 2026-03-30 06:55:40,679 - lava_callback - DEBUG - Processing node 69c77b6429015d0f95993182: baseline-arm-clabbe 2026-03-30 06:55:40,679 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77b6429015d0f95993182 2026-03-30 06:55:40,680 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:55:40,717 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77b6429015d0f95993182 HTTP/1.1" 200 None 2026-03-30 06:55:40,719 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:55:40,731 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e3a29015d0f9598f73c HTTP/1.1" 200 None 2026-03-30 06:55:40,733 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:55:40,753 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77b6429015d0f95993182 HTTP/1.1" 200 None 2026-03-30 06:55:40,758 - lava_callback - ERROR - Error processing callback for node 69c77b6429015d0f95993182: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48276 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39942 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49218 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40168 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40184 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37866 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37870 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33182 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44160 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58644 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:57:00,731 - lava_callback - INFO - Received callback for node 69c77bac29015d0f959931a6 INFO: 172.20.0.1:47614 - "POST /node/69c77bac29015d0f959931a6 HTTP/1.0" 202 Accepted 2026-03-30 06:57:00,736 - lava_callback - INFO - Starting async job submit for node 69c77bac29015d0f959931a6 2026-03-30 06:57:00,737 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:57:00,752 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77bac29015d0f959931a6 HTTP/1.1" 200 None 2026-03-30 06:57:00,753 - lava_callback - DEBUG - Processing node 69c77bac29015d0f959931a6: baseline-arm-clabbe 2026-03-30 06:57:00,753 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77bac29015d0f959931a6 2026-03-30 06:57:00,754 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:57:00,770 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77bac29015d0f959931a6 HTTP/1.1" 200 None 2026-03-30 06:57:00,772 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:57:00,787 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e3a29015d0f9598f73c HTTP/1.1" 200 None 2026-03-30 06:57:00,789 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:57:00,824 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77bac29015d0f959931a6 HTTP/1.1" 200 None 2026-03-30 06:57:00,826 - lava_callback - ERROR - Error processing callback for node 69c77bac29015d0f959931a6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47630 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35266 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35282 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33176 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33188 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56796 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:57:40,714 - lava_callback - INFO - Received callback for node 69c77bd229015d0f959931bf INFO: 172.20.0.1:43022 - "POST /node/69c77bd229015d0f959931bf HTTP/1.0" 202 Accepted 2026-03-30 06:57:40,720 - lava_callback - INFO - Starting async job submit for node 69c77bd229015d0f959931bf 2026-03-30 06:57:40,721 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:57:40,734 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77bd229015d0f959931bf HTTP/1.1" 200 None 2026-03-30 06:57:40,735 - lava_callback - DEBUG - Processing node 69c77bd229015d0f959931bf: baseline-arm-clabbe 2026-03-30 06:57:40,735 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77bd229015d0f959931bf 2026-03-30 06:57:40,736 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:57:40,748 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77bd229015d0f959931bf HTTP/1.1" 200 None 2026-03-30 06:57:40,749 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:57:40,763 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e3a29015d0f9598f73c HTTP/1.1" 200 None 2026-03-30 06:57:40,764 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:57:40,784 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77bd229015d0f959931bf HTTP/1.1" 200 None 2026-03-30 06:57:40,785 - lava_callback - ERROR - Error processing callback for node 69c77bd229015d0f959931bf: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43034 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54176 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43798 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36658 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36674 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:58:20,767 - lava_callback - INFO - Received callback for node 69c77bf729015d0f959931c9 INFO: 172.20.0.1:54824 - "POST /node/69c77bf729015d0f959931c9 HTTP/1.0" 202 Accepted 2026-03-30 06:58:20,772 - lava_callback - INFO - Starting async job submit for node 69c77bf729015d0f959931c9 2026-03-30 06:58:20,773 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:58:20,787 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77bf729015d0f959931c9 HTTP/1.1" 200 None 2026-03-30 06:58:20,788 - lava_callback - DEBUG - Processing node 69c77bf729015d0f959931c9: baseline-arm-clabbe 2026-03-30 06:58:20,788 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77bf729015d0f959931c9 2026-03-30 06:58:20,789 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:58:20,802 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77bf729015d0f959931c9 HTTP/1.1" 200 None 2026-03-30 06:58:20,804 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:58:20,817 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e3a29015d0f9598f73c HTTP/1.1" 200 None 2026-03-30 06:58:20,818 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:58:20,840 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77bf729015d0f959931c9 HTTP/1.1" 200 None 2026-03-30 06:58:20,845 - lava_callback - ERROR - Error processing callback for node 69c77bf729015d0f959931c9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54838 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54854 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60656 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43208 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46226 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:59:00,767 - lava_callback - INFO - Received callback for node 69c77c1e29015d0f959931db INFO: 172.20.0.1:49424 - "POST /node/69c77c1e29015d0f959931db HTTP/1.0" 202 Accepted 2026-03-30 06:59:00,773 - lava_callback - INFO - Starting async job submit for node 69c77c1e29015d0f959931db 2026-03-30 06:59:00,774 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:59:00,788 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77c1e29015d0f959931db HTTP/1.1" 200 None 2026-03-30 06:59:00,789 - lava_callback - DEBUG - Processing node 69c77c1e29015d0f959931db: baseline-arm-clabbe 2026-03-30 06:59:00,790 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77c1e29015d0f959931db 2026-03-30 06:59:00,790 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:59:00,810 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77c1e29015d0f959931db HTTP/1.1" 200 None 2026-03-30 06:59:00,811 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:59:00,829 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e3a29015d0f9598f73c HTTP/1.1" 200 None 2026-03-30 06:59:00,831 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:59:00,852 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77c1e29015d0f959931db HTTP/1.1" 200 None 2026-03-30 06:59:00,854 - lava_callback - ERROR - Error processing callback for node 69c77c1e29015d0f959931db: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49436 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38672 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:38678 - "GET / HTTP/1.0" 200 OK 2026-03-30 06:59:20,840 - lava_callback - INFO - Received callback for node 69c77c4329015d0f959931ef INFO: 172.20.0.1:50594 - "POST /node/69c77c4329015d0f959931ef HTTP/1.0" 202 Accepted 2026-03-30 06:59:20,846 - lava_callback - INFO - Starting async job submit for node 69c77c4329015d0f959931ef 2026-03-30 06:59:20,847 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:59:20,864 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77c4329015d0f959931ef HTTP/1.1" 200 None 2026-03-30 06:59:20,865 - lava_callback - DEBUG - Processing node 69c77c4329015d0f959931ef: baseline-arm-clabbe 2026-03-30 06:59:20,865 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77c4329015d0f959931ef 2026-03-30 06:59:20,866 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:59:20,888 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77c4329015d0f959931ef HTTP/1.1" 200 None 2026-03-30 06:59:20,889 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:59:20,904 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 06:59:20,906 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 06:59:20,926 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77c4329015d0f959931ef HTTP/1.1" 200 None 2026-03-30 06:59:20,931 - lava_callback - ERROR - Error processing callback for node 69c77c4329015d0f959931ef: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:50596 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50608 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41756 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34416 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36906 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38264 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34998 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35008 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36580 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36590 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54460 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:00:40,917 - lava_callback - INFO - Received callback for node 69c77c8c29015d0f9599321b INFO: 172.20.0.1:52638 - "POST /node/69c77c8c29015d0f9599321b HTTP/1.0" 202 Accepted 2026-03-30 07:00:40,922 - lava_callback - INFO - Starting async job submit for node 69c77c8c29015d0f9599321b 2026-03-30 07:00:40,923 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:00:40,948 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77c8c29015d0f9599321b HTTP/1.1" 200 None 2026-03-30 07:00:40,949 - lava_callback - DEBUG - Processing node 69c77c8c29015d0f9599321b: baseline-arm-clabbe 2026-03-30 07:00:40,949 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77c8c29015d0f9599321b 2026-03-30 07:00:40,950 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:00:40,962 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77c8c29015d0f9599321b HTTP/1.1" 200 None 2026-03-30 07:00:40,964 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:00:40,977 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 07:00:40,978 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:00:40,998 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77c8c29015d0f9599321b HTTP/1.1" 200 None 2026-03-30 07:00:41,000 - lava_callback - ERROR - Error processing callback for node 69c77c8c29015d0f9599321b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:52650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43514 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51844 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39678 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39686 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33152 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33168 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51030 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43540 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44084 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:02:00,935 - lava_callback - INFO - Received callback for node 69c77cd429015d0f95993237 INFO: 172.20.0.1:60870 - "POST /node/69c77cd429015d0f95993237 HTTP/1.0" 202 Accepted 2026-03-30 07:02:00,956 - lava_callback - INFO - Starting async job submit for node 69c77cd429015d0f95993237 2026-03-30 07:02:00,957 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:02:00,997 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77cd429015d0f95993237 HTTP/1.1" 200 None 2026-03-30 07:02:00,998 - lava_callback - DEBUG - Processing node 69c77cd429015d0f95993237: baseline-arm-clabbe 2026-03-30 07:02:00,999 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77cd429015d0f95993237 2026-03-30 07:02:00,999 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:02:01,015 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77cd429015d0f95993237 HTTP/1.1" 200 None 2026-03-30 07:02:01,016 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:02:01,041 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 07:02:01,043 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:02:01,063 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77cd429015d0f95993237 HTTP/1.1" 200 None 2026-03-30 07:02:01,064 - lava_callback - ERROR - Error processing callback for node 69c77cd429015d0f95993237: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:60886 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43282 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43290 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:02:20,911 - lava_callback - INFO - Received callback for node 69c77cfa29015d0f95993249 INFO: 172.20.0.1:60588 - "POST /node/69c77cfa29015d0f95993249 HTTP/1.0" 202 Accepted 2026-03-30 07:02:20,917 - lava_callback - INFO - Starting async job submit for node 69c77cfa29015d0f95993249 2026-03-30 07:02:20,918 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:02:20,933 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77cfa29015d0f95993249 HTTP/1.1" 200 None 2026-03-30 07:02:20,934 - lava_callback - DEBUG - Processing node 69c77cfa29015d0f95993249: baseline-arm-clabbe 2026-03-30 07:02:20,934 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77cfa29015d0f95993249 2026-03-30 07:02:20,935 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:02:20,949 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77cfa29015d0f95993249 HTTP/1.1" 200 None 2026-03-30 07:02:20,950 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:02:20,964 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 07:02:20,966 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:02:20,987 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77cfa29015d0f95993249 HTTP/1.1" 200 None 2026-03-30 07:02:20,990 - lava_callback - ERROR - Error processing callback for node 69c77cfa29015d0f95993249: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:60602 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60614 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50652 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60442 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48564 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:03:00,984 - lava_callback - INFO - Received callback for node 69c77d1f29015d0f9599325f INFO: 172.20.0.1:54546 - "POST /node/69c77d1f29015d0f9599325f HTTP/1.0" 202 Accepted 2026-03-30 07:03:00,989 - lava_callback - INFO - Starting async job submit for node 69c77d1f29015d0f9599325f 2026-03-30 07:03:00,990 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:03:01,012 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77d1f29015d0f9599325f HTTP/1.1" 200 None 2026-03-30 07:03:01,012 - lava_callback - DEBUG - Processing node 69c77d1f29015d0f9599325f: baseline-arm-clabbe 2026-03-30 07:03:01,013 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77d1f29015d0f9599325f 2026-03-30 07:03:01,013 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:03:01,028 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77d1f29015d0f9599325f HTTP/1.1" 200 None 2026-03-30 07:03:01,029 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:03:01,043 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 07:03:01,044 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:03:01,064 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77d1f29015d0f9599325f HTTP/1.1" 200 None 2026-03-30 07:03:01,070 - lava_callback - ERROR - Error processing callback for node 69c77d1f29015d0f9599325f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54554 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57670 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57674 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38076 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38090 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44526 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:03:40,973 - lava_callback - INFO - Received callback for node 69c77d4429015d0f9599326c INFO: 172.20.0.1:47352 - "POST /node/69c77d4429015d0f9599326c HTTP/1.0" 202 Accepted 2026-03-30 07:03:40,978 - lava_callback - INFO - Starting async job submit for node 69c77d4429015d0f9599326c 2026-03-30 07:03:40,979 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:03:41,000 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77d4429015d0f9599326c HTTP/1.1" 200 None 2026-03-30 07:03:41,001 - lava_callback - DEBUG - Processing node 69c77d4429015d0f9599326c: baseline-arm-clabbe 2026-03-30 07:03:41,001 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77d4429015d0f9599326c 2026-03-30 07:03:41,002 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:03:41,015 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77d4429015d0f9599326c HTTP/1.1" 200 None 2026-03-30 07:03:41,016 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:03:41,030 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 07:03:41,031 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:03:41,052 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77d4429015d0f9599326c HTTP/1.1" 200 None 2026-03-30 07:03:41,053 - lava_callback - ERROR - Error processing callback for node 69c77d4429015d0f9599326c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47358 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58814 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32966 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48752 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48768 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:04:21,066 - lava_callback - INFO - Received callback for node 69c77d6929015d0f9599327c INFO: 172.20.0.1:38312 - "POST /node/69c77d6929015d0f9599327c HTTP/1.0" 202 Accepted 2026-03-30 07:04:21,072 - lava_callback - INFO - Starting async job submit for node 69c77d6929015d0f9599327c 2026-03-30 07:04:21,073 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:04:21,087 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77d6929015d0f9599327c HTTP/1.1" 200 None 2026-03-30 07:04:21,088 - lava_callback - DEBUG - Processing node 69c77d6929015d0f9599327c: baseline-arm-clabbe 2026-03-30 07:04:21,088 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77d6929015d0f9599327c 2026-03-30 07:04:21,088 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:04:21,101 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77d6929015d0f9599327c HTTP/1.1" 200 None 2026-03-30 07:04:21,103 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:04:21,116 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 07:04:21,117 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:04:21,137 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77d6929015d0f9599327c HTTP/1.1" 200 None 2026-03-30 07:04:21,139 - lava_callback - ERROR - Error processing callback for node 69c77d6929015d0f9599327c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38318 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38328 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37798 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33210 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60796 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:05:01,003 - lava_callback - INFO - Received callback for node 69c77d8e29015d0f9599328c INFO: 172.20.0.1:47556 - "POST /node/69c77d8e29015d0f9599328c HTTP/1.0" 202 Accepted 2026-03-30 07:05:01,009 - lava_callback - INFO - Starting async job submit for node 69c77d8e29015d0f9599328c 2026-03-30 07:05:01,010 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:05:01,027 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77d8e29015d0f9599328c HTTP/1.1" 200 None 2026-03-30 07:05:01,028 - lava_callback - DEBUG - Processing node 69c77d8e29015d0f9599328c: baseline-arm-clabbe 2026-03-30 07:05:01,028 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77d8e29015d0f9599328c 2026-03-30 07:05:01,029 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:05:01,042 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77d8e29015d0f9599328c HTTP/1.1" 200 None 2026-03-30 07:05:01,044 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:05:01,064 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 07:05:01,066 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:05:01,086 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77d8e29015d0f9599328c HTTP/1.1" 200 None 2026-03-30 07:05:01,092 - lava_callback - ERROR - Error processing callback for node 69c77d8e29015d0f9599328c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47562 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53770 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53782 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56664 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51588 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:05:41,076 - lava_callback - INFO - Received callback for node 69c77db329015d0f959932a6 INFO: 172.20.0.1:47212 - "POST /node/69c77db329015d0f959932a6 HTTP/1.0" 202 Accepted 2026-03-30 07:05:41,083 - lava_callback - INFO - Starting async job submit for node 69c77db329015d0f959932a6 2026-03-30 07:05:41,084 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:05:41,101 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77db329015d0f959932a6 HTTP/1.1" 200 None 2026-03-30 07:05:41,103 - lava_callback - DEBUG - Processing node 69c77db329015d0f959932a6: baseline-arm-clabbe 2026-03-30 07:05:41,103 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77db329015d0f959932a6 2026-03-30 07:05:41,104 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:05:41,120 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77db329015d0f959932a6 HTTP/1.1" 200 None 2026-03-30 07:05:41,121 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:05:41,151 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 07:05:41,152 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:05:41,174 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77db329015d0f959932a6 HTTP/1.1" 200 None 2026-03-30 07:05:41,178 - lava_callback - ERROR - Error processing callback for node 69c77db329015d0f959932a6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47222 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50780 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56954 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45934 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45942 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:06:21,085 - lava_callback - INFO - Received callback for node 69c77dd829015d0f959932b4 INFO: 172.20.0.1:47730 - "POST /node/69c77dd829015d0f959932b4 HTTP/1.0" 202 Accepted 2026-03-30 07:06:21,090 - lava_callback - INFO - Starting async job submit for node 69c77dd829015d0f959932b4 2026-03-30 07:06:21,091 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:06:21,105 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77dd829015d0f959932b4 HTTP/1.1" 200 None 2026-03-30 07:06:21,106 - lava_callback - DEBUG - Processing node 69c77dd829015d0f959932b4: baseline-arm-clabbe 2026-03-30 07:06:21,107 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77dd829015d0f959932b4 2026-03-30 07:06:21,107 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:06:21,120 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77dd829015d0f959932b4 HTTP/1.1" 200 None 2026-03-30 07:06:21,121 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:06:21,134 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 07:06:21,136 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:06:21,166 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77dd829015d0f959932b4 HTTP/1.1" 200 None 2026-03-30 07:06:21,168 - lava_callback - ERROR - Error processing callback for node 69c77dd829015d0f959932b4: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47744 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47758 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46464 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:06:41,150 - lava_callback - INFO - Received callback for node 69c77dfd29015d0f959932c5 INFO: 172.20.0.1:37644 - "POST /node/69c77dfd29015d0f959932c5 HTTP/1.0" 202 Accepted 2026-03-30 07:06:41,156 - lava_callback - INFO - Starting async job submit for node 69c77dfd29015d0f959932c5 2026-03-30 07:06:41,157 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:06:41,179 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77dfd29015d0f959932c5 HTTP/1.1" 200 None 2026-03-30 07:06:41,180 - lava_callback - DEBUG - Processing node 69c77dfd29015d0f959932c5: baseline-arm-clabbe 2026-03-30 07:06:41,181 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77dfd29015d0f959932c5 2026-03-30 07:06:41,181 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:06:41,194 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77dfd29015d0f959932c5 HTTP/1.1" 200 None 2026-03-30 07:06:41,195 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:06:41,209 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 07:06:41,211 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:06:41,231 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77dfd29015d0f959932c5 HTTP/1.1" 200 None 2026-03-30 07:06:41,233 - lava_callback - ERROR - Error processing callback for node 69c77dfd29015d0f959932c5: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:37656 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40352 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42208 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39268 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39270 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:07:21,128 - lava_callback - INFO - Received callback for node 69c77e2229015d0f959932d5 INFO: 172.20.0.1:53730 - "POST /node/69c77e2229015d0f959932d5 HTTP/1.0" 202 Accepted 2026-03-30 07:07:21,133 - lava_callback - INFO - Starting async job submit for node 69c77e2229015d0f959932d5 2026-03-30 07:07:21,134 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:07:21,168 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77e2229015d0f959932d5 HTTP/1.1" 200 None 2026-03-30 07:07:21,169 - lava_callback - DEBUG - Processing node 69c77e2229015d0f959932d5: baseline-arm-clabbe 2026-03-30 07:07:21,169 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77e2229015d0f959932d5 2026-03-30 07:07:21,170 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:07:21,183 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77e2229015d0f959932d5 HTTP/1.1" 200 None 2026-03-30 07:07:21,185 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:07:21,198 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b3b590270ad7b6d698b HTTP/1.1" 200 None 2026-03-30 07:07:21,199 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:07:21,220 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77e2229015d0f959932d5 HTTP/1.1" 200 None 2026-03-30 07:07:21,229 - lava_callback - ERROR - Error processing callback for node 69c77e2229015d0f959932d5: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:53736 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53742 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32854 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59952 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:08:01,160 - lava_callback - INFO - Received callback for node 69c77e4829015d0f959932ee INFO: 172.20.0.1:56876 - "POST /node/69c77e4829015d0f959932ee HTTP/1.0" 202 Accepted 2026-03-30 07:08:01,166 - lava_callback - INFO - Starting async job submit for node 69c77e4829015d0f959932ee 2026-03-30 07:08:01,167 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:08:01,192 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77e4829015d0f959932ee HTTP/1.1" 200 None 2026-03-30 07:08:01,193 - lava_callback - DEBUG - Processing node 69c77e4829015d0f959932ee: baseline-arm-clabbe 2026-03-30 07:08:01,193 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77e4829015d0f959932ee 2026-03-30 07:08:01,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:08:01,208 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77e4829015d0f959932ee HTTP/1.1" 200 None 2026-03-30 07:08:01,209 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:08:01,222 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 07:08:01,223 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:08:01,243 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77e4829015d0f959932ee HTTP/1.1" 200 None 2026-03-30 07:08:01,248 - lava_callback - ERROR - Error processing callback for node 69c77e4829015d0f959932ee: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:56890 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40424 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40434 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34476 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34488 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48000 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:08:41,160 - lava_callback - INFO - Received callback for node 69c77e6d29015d0f959932f6 INFO: 172.20.0.1:45638 - "POST /node/69c77e6d29015d0f959932f6 HTTP/1.0" 202 Accepted 2026-03-30 07:08:41,165 - lava_callback - INFO - Starting async job submit for node 69c77e6d29015d0f959932f6 2026-03-30 07:08:41,166 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:08:41,180 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77e6d29015d0f959932f6 HTTP/1.1" 200 None 2026-03-30 07:08:41,181 - lava_callback - DEBUG - Processing node 69c77e6d29015d0f959932f6: baseline-arm-clabbe 2026-03-30 07:08:41,181 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77e6d29015d0f959932f6 2026-03-30 07:08:41,182 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:08:41,195 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77e6d29015d0f959932f6 HTTP/1.1" 200 None 2026-03-30 07:08:41,197 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:08:41,209 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 07:08:41,211 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:08:41,236 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77e6d29015d0f959932f6 HTTP/1.1" 200 None 2026-03-30 07:08:41,239 - lava_callback - ERROR - Error processing callback for node 69c77e6d29015d0f959932f6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32868 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44996 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48002 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48014 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:09:21,238 - lava_callback - INFO - Received callback for node 69c77e9229015d0f95993306 INFO: 172.20.0.1:43138 - "POST /node/69c77e9229015d0f95993306 HTTP/1.0" 202 Accepted 2026-03-30 07:09:21,244 - lava_callback - INFO - Starting async job submit for node 69c77e9229015d0f95993306 2026-03-30 07:09:21,245 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:09:21,259 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77e9229015d0f95993306 HTTP/1.1" 200 None 2026-03-30 07:09:21,259 - lava_callback - DEBUG - Processing node 69c77e9229015d0f95993306: baseline-arm-clabbe 2026-03-30 07:09:21,260 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77e9229015d0f95993306 2026-03-30 07:09:21,260 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:09:21,274 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77e9229015d0f95993306 HTTP/1.1" 200 None 2026-03-30 07:09:21,276 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:09:21,289 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 07:09:21,290 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:09:21,310 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77e9229015d0f95993306 HTTP/1.1" 200 None 2026-03-30 07:09:21,311 - lava_callback - ERROR - Error processing callback for node 69c77e9229015d0f95993306: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:43142 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43150 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34268 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38790 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54748 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:10:01,179 - lava_callback - INFO - Received callback for node 69c77eb729015d0f95993316 INFO: 172.20.0.1:53012 - "POST /node/69c77eb729015d0f95993316 HTTP/1.0" 202 Accepted 2026-03-30 07:10:01,185 - lava_callback - INFO - Starting async job submit for node 69c77eb729015d0f95993316 2026-03-30 07:10:01,186 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:10:01,213 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77eb729015d0f95993316 HTTP/1.1" 200 None 2026-03-30 07:10:01,213 - lava_callback - DEBUG - Processing node 69c77eb729015d0f95993316: baseline-arm-clabbe 2026-03-30 07:10:01,214 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77eb729015d0f95993316 2026-03-30 07:10:01,214 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:10:01,228 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77eb729015d0f95993316 HTTP/1.1" 200 None 2026-03-30 07:10:01,229 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:10:01,256 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 07:10:01,257 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:10:01,278 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77eb729015d0f95993316 HTTP/1.1" 200 None 2026-03-30 07:10:01,283 - lava_callback - ERROR - Error processing callback for node 69c77eb729015d0f95993316: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:53028 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47784 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47788 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60012 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60026 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40482 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:10:41,279 - lava_callback - INFO - Received callback for node 69c77edc29015d0f95993326 INFO: 172.20.0.1:38146 - "POST /node/69c77edc29015d0f95993326 HTTP/1.0" 202 Accepted 2026-03-30 07:10:41,285 - lava_callback - INFO - Starting async job submit for node 69c77edc29015d0f95993326 2026-03-30 07:10:41,286 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:10:41,301 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77edc29015d0f95993326 HTTP/1.1" 200 None 2026-03-30 07:10:41,302 - lava_callback - DEBUG - Processing node 69c77edc29015d0f95993326: baseline-arm-clabbe 2026-03-30 07:10:41,302 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77edc29015d0f95993326 2026-03-30 07:10:41,303 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:10:41,316 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77edc29015d0f95993326 HTTP/1.1" 200 None 2026-03-30 07:10:41,317 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:10:41,331 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 07:10:41,332 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:10:41,353 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77edc29015d0f95993326 HTTP/1.1" 200 None 2026-03-30 07:10:41,358 - lava_callback - ERROR - Error processing callback for node 69c77edc29015d0f95993326: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38148 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51854 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:11:01,313 - lava_callback - INFO - Received callback for node 69c77f0129015d0f95993338 INFO: 172.20.0.1:51086 - "POST /node/69c77f0129015d0f95993338 HTTP/1.0" 202 Accepted 2026-03-30 07:11:01,318 - lava_callback - INFO - Starting async job submit for node 69c77f0129015d0f95993338 2026-03-30 07:11:01,319 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:11:01,334 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77f0129015d0f95993338 HTTP/1.1" 200 None 2026-03-30 07:11:01,334 - lava_callback - DEBUG - Processing node 69c77f0129015d0f95993338: baseline-arm-clabbe 2026-03-30 07:11:01,335 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77f0129015d0f95993338 2026-03-30 07:11:01,335 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:11:01,349 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77f0129015d0f95993338 HTTP/1.1" 200 None 2026-03-30 07:11:01,351 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:11:01,364 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 07:11:01,365 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:11:01,397 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77f0129015d0f95993338 HTTP/1.1" 200 None 2026-03-30 07:11:01,403 - lava_callback - ERROR - Error processing callback for node 69c77f0129015d0f95993338: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:51096 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36484 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:36490 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58986 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59000 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40706 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:11:41,343 - lava_callback - INFO - Received callback for node 69c77f2629015d0f95993348 INFO: 172.20.0.1:53894 - "POST /node/69c77f2629015d0f95993348 HTTP/1.0" 202 Accepted 2026-03-30 07:11:41,349 - lava_callback - INFO - Starting async job submit for node 69c77f2629015d0f95993348 2026-03-30 07:11:41,350 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:11:41,366 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77f2629015d0f95993348 HTTP/1.1" 200 None 2026-03-30 07:11:41,367 - lava_callback - DEBUG - Processing node 69c77f2629015d0f95993348: baseline-arm-clabbe 2026-03-30 07:11:41,368 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77f2629015d0f95993348 2026-03-30 07:11:41,369 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:11:41,384 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77f2629015d0f95993348 HTTP/1.1" 200 None 2026-03-30 07:11:41,386 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:11:41,409 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 07:11:41,411 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:11:41,431 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77f2629015d0f95993348 HTTP/1.1" 200 None 2026-03-30 07:11:41,432 - lava_callback - ERROR - Error processing callback for node 69c77f2629015d0f95993348: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:53906 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41260 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56522 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44954 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44964 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:12:21,373 - lava_callback - INFO - Received callback for node 69c77f4b29015d0f95993359 INFO: 172.20.0.1:50840 - "POST /node/69c77f4b29015d0f95993359 HTTP/1.0" 202 Accepted 2026-03-30 07:12:21,379 - lava_callback - INFO - Starting async job submit for node 69c77f4b29015d0f95993359 2026-03-30 07:12:21,380 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:12:21,400 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77f4b29015d0f95993359 HTTP/1.1" 200 None 2026-03-30 07:12:21,401 - lava_callback - DEBUG - Processing node 69c77f4b29015d0f95993359: baseline-arm-clabbe 2026-03-30 07:12:21,401 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77f4b29015d0f95993359 2026-03-30 07:12:21,401 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:12:21,416 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77f4b29015d0f95993359 HTTP/1.1" 200 None 2026-03-30 07:12:21,417 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:12:21,431 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 07:12:21,432 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:12:21,453 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77f4b29015d0f95993359 HTTP/1.1" 200 None 2026-03-30 07:12:21,454 - lava_callback - ERROR - Error processing callback for node 69c77f4b29015d0f95993359: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:50842 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50858 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59264 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37420 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46160 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:13:01,448 - lava_callback - INFO - Received callback for node 69c77f7029015d0f95993368 INFO: 172.20.0.1:47342 - "POST /node/69c77f7029015d0f95993368 HTTP/1.0" 202 Accepted 2026-03-30 07:13:01,454 - lava_callback - INFO - Starting async job submit for node 69c77f7029015d0f95993368 2026-03-30 07:13:01,455 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:13:01,469 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77f7029015d0f95993368 HTTP/1.1" 200 None 2026-03-30 07:13:01,470 - lava_callback - DEBUG - Processing node 69c77f7029015d0f95993368: baseline-arm-clabbe 2026-03-30 07:13:01,470 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77f7029015d0f95993368 2026-03-30 07:13:01,471 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:13:01,484 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77f7029015d0f95993368 HTTP/1.1" 200 None 2026-03-30 07:13:01,486 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:13:01,500 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 07:13:01,502 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:13:01,527 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77f7029015d0f95993368 HTTP/1.1" 200 None 2026-03-30 07:13:01,528 - lava_callback - ERROR - Error processing callback for node 69c77f7029015d0f95993368: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47346 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47320 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47328 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37744 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37748 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36620 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:13:41,459 - lava_callback - INFO - Received callback for node 69c77f9529015d0f9599337d INFO: 172.20.0.1:45004 - "POST /node/69c77f9529015d0f9599337d HTTP/1.0" 202 Accepted 2026-03-30 07:13:41,465 - lava_callback - INFO - Starting async job submit for node 69c77f9529015d0f9599337d 2026-03-30 07:13:41,466 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:13:41,481 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77f9529015d0f9599337d HTTP/1.1" 200 None 2026-03-30 07:13:41,482 - lava_callback - DEBUG - Processing node 69c77f9529015d0f9599337d: baseline-arm-clabbe 2026-03-30 07:13:41,482 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77f9529015d0f9599337d 2026-03-30 07:13:41,483 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:13:41,497 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77f9529015d0f9599337d HTTP/1.1" 200 None 2026-03-30 07:13:41,499 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:13:41,513 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 07:13:41,515 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:13:41,535 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77f9529015d0f9599337d HTTP/1.1" 200 None 2026-03-30 07:13:41,537 - lava_callback - ERROR - Error processing callback for node 69c77f9529015d0f9599337d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:45018 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51382 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51900 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59120 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59122 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:14:21,455 - lava_callback - INFO - Received callback for node 69c77fbb29015d0f9599338a INFO: 172.20.0.1:39036 - "POST /node/69c77fbb29015d0f9599338a HTTP/1.0" 202 Accepted 2026-03-30 07:14:21,460 - lava_callback - INFO - Starting async job submit for node 69c77fbb29015d0f9599338a 2026-03-30 07:14:21,461 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:14:21,476 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77fbb29015d0f9599338a HTTP/1.1" 200 None 2026-03-30 07:14:21,477 - lava_callback - DEBUG - Processing node 69c77fbb29015d0f9599338a: baseline-arm-clabbe 2026-03-30 07:14:21,477 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77fbb29015d0f9599338a 2026-03-30 07:14:21,478 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:14:21,500 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77fbb29015d0f9599338a HTTP/1.1" 200 None 2026-03-30 07:14:21,502 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:14:21,515 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 07:14:21,517 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:14:21,536 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77fbb29015d0f9599338a HTTP/1.1" 200 None 2026-03-30 07:14:21,542 - lava_callback - ERROR - Error processing callback for node 69c77fbb29015d0f9599338a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:39050 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39052 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56460 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49950 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53398 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:15:01,413 - lava_callback - INFO - Received callback for node 69c77fe029015d0f95993399 INFO: 172.20.0.1:56152 - "POST /node/69c77fe029015d0f95993399 HTTP/1.0" 202 Accepted 2026-03-30 07:15:01,420 - lava_callback - INFO - Starting async job submit for node 69c77fe029015d0f95993399 2026-03-30 07:15:01,421 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:15:01,439 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77fe029015d0f95993399 HTTP/1.1" 200 None 2026-03-30 07:15:01,440 - lava_callback - DEBUG - Processing node 69c77fe029015d0f95993399: baseline-arm-clabbe 2026-03-30 07:15:01,440 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c77fe029015d0f95993399 2026-03-30 07:15:01,441 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:15:01,457 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c77fe029015d0f95993399 HTTP/1.1" 200 None 2026-03-30 07:15:01,458 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:15:01,472 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 07:15:01,474 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:15:01,527 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c77fe029015d0f95993399 HTTP/1.1" 200 None 2026-03-30 07:15:01,529 - lava_callback - ERROR - Error processing callback for node 69c77fe029015d0f95993399: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:56156 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39648 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39650 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:15:21,492 - lava_callback - INFO - Received callback for node 69c7800529015d0f959933ac INFO: 172.20.0.1:41476 - "POST /node/69c7800529015d0f959933ac HTTP/1.0" 202 Accepted 2026-03-30 07:15:21,497 - lava_callback - INFO - Starting async job submit for node 69c7800529015d0f959933ac 2026-03-30 07:15:21,498 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:15:21,513 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7800529015d0f959933ac HTTP/1.1" 200 None 2026-03-30 07:15:21,514 - lava_callback - DEBUG - Processing node 69c7800529015d0f959933ac: baseline-arm-clabbe 2026-03-30 07:15:21,514 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7800529015d0f959933ac 2026-03-30 07:15:21,515 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:15:21,527 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7800529015d0f959933ac HTTP/1.1" 200 None 2026-03-30 07:15:21,528 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:15:21,542 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d82b0efaf21d93103543 HTTP/1.1" 200 None 2026-03-30 07:15:21,543 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:15:21,594 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7800529015d0f959933ac HTTP/1.1" 200 None 2026-03-30 07:15:21,595 - lava_callback - ERROR - Error processing callback for node 69c7800529015d0f959933ac: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41478 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41488 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42528 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49566 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48092 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41336 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46026 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46040 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45668 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45676 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37860 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55428 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36290 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42604 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50228 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50236 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50114 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50130 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36482 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:17:43,790 - lava_callback - INFO - Received callback for node 69c7808429015d0f959933ce INFO: 172.20.0.1:37120 - "POST /node/69c7808429015d0f959933ce HTTP/1.0" 202 Accepted 2026-03-30 07:17:43,796 - lava_callback - INFO - Starting async job submit for node 69c7808429015d0f959933ce 2026-03-30 07:17:43,797 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:17:43,811 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7808429015d0f959933ce HTTP/1.1" 200 None 2026-03-30 07:17:43,812 - lava_callback - DEBUG - Processing node 69c7808429015d0f959933ce: baseline-arm64-baylibre 2026-03-30 07:17:43,812 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7808429015d0f959933ce 2026-03-30 07:17:43,813 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:17:43,826 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7808429015d0f959933ce HTTP/1.1" 200 None 2026-03-30 07:17:43,829 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:17:43,848 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 07:17:43,849 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:17:43,870 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7808429015d0f959933ce HTTP/1.1" 200 None 2026-03-30 07:17:43,879 - lava_callback - ERROR - Error processing callback for node 69c7808429015d0f959933ce: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:37128 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59284 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47514 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53454 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53460 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33526 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33542 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59922 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54396 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59168 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:19:03,721 - lava_callback - INFO - Received callback for node 69c780d529015d0f959933d6 INFO: 172.20.0.1:49002 - "POST /node/69c780d529015d0f959933d6 HTTP/1.0" 202 Accepted 2026-03-30 07:19:03,727 - lava_callback - INFO - Starting async job submit for node 69c780d529015d0f959933d6 2026-03-30 07:19:03,727 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:19:03,742 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c780d529015d0f959933d6 HTTP/1.1" 200 None 2026-03-30 07:19:03,742 - lava_callback - DEBUG - Processing node 69c780d529015d0f959933d6: baseline-arm64-baylibre 2026-03-30 07:19:03,743 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c780d529015d0f959933d6 2026-03-30 07:19:03,743 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:19:03,757 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c780d529015d0f959933d6 HTTP/1.1" 200 None 2026-03-30 07:19:03,758 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:19:03,772 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 07:19:03,773 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:19:03,804 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c780d529015d0f959933d6 HTTP/1.1" 200 None 2026-03-30 07:19:03,806 - lava_callback - ERROR - Error processing callback for node 69c780d529015d0f959933d6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49014 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39712 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39716 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39114 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39118 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36090 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55766 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58874 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34362 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47022 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47028 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:20:23,900 - lava_callback - INFO - Received callback for node 69c7812a29015d0f959933e2 INFO: 172.20.0.1:58866 - "POST /node/69c7812a29015d0f959933e2 HTTP/1.0" 202 Accepted 2026-03-30 07:20:23,906 - lava_callback - INFO - Starting async job submit for node 69c7812a29015d0f959933e2 INFO: 172.20.0.1:58880 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:20:23,907 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:20:23,922 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7812a29015d0f959933e2 HTTP/1.1" 200 None 2026-03-30 07:20:23,923 - lava_callback - DEBUG - Processing node 69c7812a29015d0f959933e2: baseline-arm64-baylibre 2026-03-30 07:20:23,923 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7812a29015d0f959933e2 2026-03-30 07:20:23,924 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:20:23,936 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7812a29015d0f959933e2 HTTP/1.1" 200 None 2026-03-30 07:20:23,938 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:20:23,951 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 07:20:23,952 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:20:23,974 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7812a29015d0f959933e2 HTTP/1.1" 200 None 2026-03-30 07:20:23,981 - lava_callback - ERROR - Error processing callback for node 69c7812a29015d0f959933e2: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:58882 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46506 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41444 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54564 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55478 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42442 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42446 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55424 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55440 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46108 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:21:43,906 - lava_callback - INFO - Received callback for node 69c7817c29015d0f959933f9 INFO: 172.20.0.1:48362 - "POST /node/69c7817c29015d0f959933f9 HTTP/1.0" 202 Accepted 2026-03-30 07:21:43,915 - lava_callback - INFO - Starting async job submit for node 69c7817c29015d0f959933f9 2026-03-30 07:21:43,916 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:21:43,931 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7817c29015d0f959933f9 HTTP/1.1" 200 None 2026-03-30 07:21:43,932 - lava_callback - DEBUG - Processing node 69c7817c29015d0f959933f9: baseline-arm64-baylibre 2026-03-30 07:21:43,932 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7817c29015d0f959933f9 2026-03-30 07:21:43,933 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:21:43,956 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7817c29015d0f959933f9 HTTP/1.1" 200 None 2026-03-30 07:21:43,958 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:21:43,975 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 07:21:43,977 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:21:44,000 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7817c29015d0f959933f9 HTTP/1.1" 200 None 2026-03-30 07:21:44,002 - lava_callback - ERROR - Error processing callback for node 69c7817c29015d0f959933f9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48372 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52466 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52126 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52136 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33312 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33320 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33326 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33338 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49700 - "GET /favicon.ico HTTP/1.0" 404 Not Found INFO: 172.20.0.1:49716 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34232 - "GET /login HTTP/1.0" 404 Not Found INFO: 172.20.0.1:34240 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53984 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:23:04,113 - lava_callback - INFO - Received callback for node 69c781d129015d0f9599342f INFO: 172.20.0.1:49062 - "POST /node/69c781d129015d0f9599342f HTTP/1.0" 202 Accepted 2026-03-30 07:23:04,119 - lava_callback - INFO - Starting async job submit for node 69c781d129015d0f9599342f 2026-03-30 07:23:04,120 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:23:04,134 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c781d129015d0f9599342f HTTP/1.1" 200 None 2026-03-30 07:23:04,135 - lava_callback - DEBUG - Processing node 69c781d129015d0f9599342f: baseline-arm64-baylibre 2026-03-30 07:23:04,135 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c781d129015d0f9599342f 2026-03-30 07:23:04,136 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:23:04,149 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c781d129015d0f9599342f HTTP/1.1" 200 None 2026-03-30 07:23:04,150 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:23:04,164 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 07:23:04,166 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:23:04,195 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c781d129015d0f9599342f HTTP/1.1" 200 None 2026-03-30 07:23:04,196 - lava_callback - ERROR - Error processing callback for node 69c781d129015d0f9599342f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49068 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58136 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58142 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47194 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47196 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52542 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42896 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37388 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38144 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41600 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41610 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32932 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53792 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41242 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47740 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55066 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40026 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:40038 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49930 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49940 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38886 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36008 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45742 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54710 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58716 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58720 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49268 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49274 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56430 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45944 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54884 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59106 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47858 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:47868 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56828 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56844 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37034 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38990 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60974 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50984 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59908 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59922 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36640 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36642 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44450 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35800 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40030 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37798 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59310 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59322 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40688 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40690 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47938 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:29:44,353 - lava_callback - INFO - Received callback for node 69c7836429015d0f9599345d INFO: 172.20.0.1:59404 - "POST /node/69c7836429015d0f9599345d HTTP/1.0" 202 Accepted 2026-03-30 07:29:44,359 - lava_callback - INFO - Starting async job submit for node 69c7836429015d0f9599345d 2026-03-30 07:29:44,360 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:29:44,378 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7836429015d0f9599345d HTTP/1.1" 200 None 2026-03-30 07:29:44,379 - lava_callback - DEBUG - Processing node 69c7836429015d0f9599345d: baseline-arm64-baylibre 2026-03-30 07:29:44,379 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7836429015d0f9599345d 2026-03-30 07:29:44,379 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:29:44,393 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7836429015d0f9599345d HTTP/1.1" 200 None 2026-03-30 07:29:44,394 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:29:44,408 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 07:29:44,410 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:29:44,431 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7836429015d0f9599345d HTTP/1.1" 200 None 2026-03-30 07:29:44,432 - lava_callback - ERROR - Error processing callback for node 69c7836429015d0f9599345d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:59414 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59146 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35590 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35596 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55098 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55110 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36262 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49016 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33186 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:31:04,388 - lava_callback - INFO - Received callback for node 69c783b429015d0f95993469 INFO: 172.20.0.1:41512 - "POST /node/69c783b429015d0f95993469 HTTP/1.0" 202 Accepted 2026-03-30 07:31:04,394 - lava_callback - INFO - Starting async job submit for node 69c783b429015d0f95993469 2026-03-30 07:31:04,395 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:31:04,422 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c783b429015d0f95993469 HTTP/1.1" 200 None 2026-03-30 07:31:04,423 - lava_callback - DEBUG - Processing node 69c783b429015d0f95993469: baseline-arm64-baylibre 2026-03-30 07:31:04,423 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c783b429015d0f95993469 2026-03-30 07:31:04,424 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:31:04,446 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c783b429015d0f95993469 HTTP/1.1" 200 None 2026-03-30 07:31:04,448 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:31:04,466 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 07:31:04,468 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:31:04,488 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c783b429015d0f95993469 HTTP/1.1" 200 None 2026-03-30 07:31:04,489 - lava_callback - ERROR - Error processing callback for node 69c783b429015d0f95993469: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41528 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45770 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45784 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60648 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60662 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51534 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46980 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36028 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39388 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33020 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33030 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40088 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:32:24,470 - lava_callback - INFO - Received callback for node 69c7840529015d0f95993473 INFO: 172.20.0.1:40092 - "POST /node/69c7840529015d0f95993473 HTTP/1.0" 202 Accepted 2026-03-30 07:32:24,476 - lava_callback - INFO - Starting async job submit for node 69c7840529015d0f95993473 2026-03-30 07:32:24,477 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:32:24,504 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7840529015d0f95993473 HTTP/1.1" 200 None 2026-03-30 07:32:24,505 - lava_callback - DEBUG - Processing node 69c7840529015d0f95993473: baseline-arm64-baylibre 2026-03-30 07:32:24,505 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7840529015d0f95993473 2026-03-30 07:32:24,506 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:32:24,523 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7840529015d0f95993473 HTTP/1.1" 200 None 2026-03-30 07:32:24,525 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:32:24,538 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 07:32:24,539 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:32:24,560 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7840529015d0f95993473 HTTP/1.1" 200 None 2026-03-30 07:32:24,561 - lava_callback - ERROR - Error processing callback for node 69c7840529015d0f95993473: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:40102 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49234 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51840 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50292 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44410 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35934 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35940 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59294 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59296 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45342 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:33:44,661 - lava_callback - INFO - Received callback for node 69c7845629015d0f95993481 INFO: 172.20.0.1:37748 - "POST /node/69c7845629015d0f95993481 HTTP/1.0" 202 Accepted 2026-03-30 07:33:44,667 - lava_callback - INFO - Starting async job submit for node 69c7845629015d0f95993481 2026-03-30 07:33:44,668 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:33:44,682 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7845629015d0f95993481 HTTP/1.1" 200 None 2026-03-30 07:33:44,683 - lava_callback - DEBUG - Processing node 69c7845629015d0f95993481: baseline-arm64-baylibre 2026-03-30 07:33:44,683 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7845629015d0f95993481 2026-03-30 07:33:44,684 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:33:44,698 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7845629015d0f95993481 HTTP/1.1" 200 None 2026-03-30 07:33:44,699 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:33:44,713 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 07:33:44,715 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:33:44,735 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7845629015d0f95993481 HTTP/1.1" 200 None 2026-03-30 07:33:44,744 - lava_callback - ERROR - Error processing callback for node 69c7845629015d0f95993481: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:37764 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43064 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36484 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50264 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:50280 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43038 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43044 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53674 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54014 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43946 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36048 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39398 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39414 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40248 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:35:24,645 - lava_callback - INFO - Received callback for node 69c784a729015d0f9599348f INFO: 172.20.0.1:40250 - "POST /node/69c784a729015d0f9599348f HTTP/1.0" 202 Accepted 2026-03-30 07:35:24,651 - lava_callback - INFO - Starting async job submit for node 69c784a729015d0f9599348f 2026-03-30 07:35:24,651 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:35:24,666 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c784a729015d0f9599348f HTTP/1.1" 200 None 2026-03-30 07:35:24,666 - lava_callback - DEBUG - Processing node 69c784a729015d0f9599348f: baseline-arm64-baylibre 2026-03-30 07:35:24,667 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c784a729015d0f9599348f 2026-03-30 07:35:24,667 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:35:24,680 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c784a729015d0f9599348f HTTP/1.1" 200 None 2026-03-30 07:35:24,682 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:35:24,708 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 07:35:24,710 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:35:24,733 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c784a729015d0f9599348f HTTP/1.1" 200 None 2026-03-30 07:35:24,734 - lava_callback - ERROR - Error processing callback for node 69c784a729015d0f9599348f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:40264 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38802 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53808 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57746 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57700 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44358 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:44366 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46998 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47014 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47016 - "GET /node/69a7f88c7c78d2166ff8612d%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.20.0.1:57808 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:36:44,701 - lava_callback - INFO - Received callback for node 69c784fb29015d0f9599349e INFO: 172.20.0.1:48208 - "POST /node/69c784fb29015d0f9599349e HTTP/1.0" 202 Accepted 2026-03-30 07:36:44,707 - lava_callback - INFO - Starting async job submit for node 69c784fb29015d0f9599349e 2026-03-30 07:36:44,707 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:36:44,722 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c784fb29015d0f9599349e HTTP/1.1" 200 None 2026-03-30 07:36:44,722 - lava_callback - DEBUG - Processing node 69c784fb29015d0f9599349e: baseline-arm64-baylibre 2026-03-30 07:36:44,723 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c784fb29015d0f9599349e 2026-03-30 07:36:44,723 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:36:44,737 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c784fb29015d0f9599349e HTTP/1.1" 200 None 2026-03-30 07:36:44,738 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:36:44,751 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 07:36:44,752 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:36:44,773 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c784fb29015d0f9599349e HTTP/1.1" 200 None 2026-03-30 07:36:44,774 - lava_callback - ERROR - Error processing callback for node 69c784fb29015d0f9599349e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48214 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51662 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50636 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43694 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43702 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53810 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53814 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48780 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49844 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48634 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52922 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58096 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:58110 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37392 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37408 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59256 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52254 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47644 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35798 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54370 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:54384 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58482 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58486 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43758 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51056 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35726 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39148 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41320 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:41336 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37706 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37712 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32768 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:40:42,700 - lava_callback - INFO - Received callback for node 69c785f329015d0f959934c7 INFO: 172.20.0.1:34848 - "POST /node/69c785f329015d0f959934c7 HTTP/1.0" 202 Accepted 2026-03-30 07:40:42,706 - lava_callback - INFO - Starting async job submit for node 69c785f329015d0f959934c7 2026-03-30 07:40:42,707 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:40:42,722 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c785f329015d0f959934c7 HTTP/1.1" 200 None 2026-03-30 07:40:42,722 - lava_callback - DEBUG - Processing node 69c785f329015d0f959934c7: baseline-arm64-clabbe 2026-03-30 07:40:42,723 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c785f329015d0f959934c7 2026-03-30 07:40:42,723 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:40:42,737 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c785f329015d0f959934c7 HTTP/1.1" 200 None 2026-03-30 07:40:42,739 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:40:42,779 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 07:40:42,780 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:40:42,808 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c785f329015d0f959934c7 HTTP/1.1" 200 None 2026-03-30 07:40:42,810 - lava_callback - ERROR - Error processing callback for node 69c785f329015d0f959934c7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:34858 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40946 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57720 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55422 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:55428 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53316 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53318 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58518 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44098 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46434 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:42:02,749 - lava_callback - INFO - Received callback for node 69c7864329015d0f959934d2 INFO: 172.20.0.1:44502 - "POST /node/69c7864329015d0f959934d2 HTTP/1.0" 202 Accepted 2026-03-30 07:42:02,755 - lava_callback - INFO - Starting async job submit for node 69c7864329015d0f959934d2 2026-03-30 07:42:02,756 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:42:02,771 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7864329015d0f959934d2 HTTP/1.1" 200 None 2026-03-30 07:42:02,772 - lava_callback - DEBUG - Processing node 69c7864329015d0f959934d2: baseline-arm64-clabbe 2026-03-30 07:42:02,772 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7864329015d0f959934d2 2026-03-30 07:42:02,773 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:42:02,802 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7864329015d0f959934d2 HTTP/1.1" 200 None 2026-03-30 07:42:02,803 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:42:02,825 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 07:42:02,826 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:42:02,847 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7864329015d0f959934d2 HTTP/1.1" 200 None 2026-03-30 07:42:02,852 - lava_callback - ERROR - Error processing callback for node 69c7864329015d0f959934d2: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44512 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46504 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46518 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34058 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34070 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58838 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42446 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60784 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43032 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42768 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42776 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:43:22,778 - lava_callback - INFO - Received callback for node 69c7869329015d0f959934df INFO: 172.20.0.1:35846 - "POST /node/69c7869329015d0f959934df HTTP/1.0" 202 Accepted 2026-03-30 07:43:22,784 - lava_callback - INFO - Starting async job submit for node 69c7869329015d0f959934df 2026-03-30 07:43:22,785 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:43:22,811 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7869329015d0f959934df HTTP/1.1" 200 None 2026-03-30 07:43:22,812 - lava_callback - DEBUG - Processing node 69c7869329015d0f959934df: baseline-arm64-clabbe 2026-03-30 07:43:22,813 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7869329015d0f959934df 2026-03-30 07:43:22,813 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:43:22,827 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7869329015d0f959934df HTTP/1.1" 200 None 2026-03-30 07:43:22,828 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:43:22,843 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 07:43:22,844 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:43:22,864 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7869329015d0f959934df HTTP/1.1" 200 None 2026-03-30 07:43:22,866 - lava_callback - ERROR - Error processing callback for node 69c7869329015d0f959934df: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:35854 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35868 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35228 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38650 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49526 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36724 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53798 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:53810 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48552 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48568 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46166 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:44:42,815 - lava_callback - INFO - Received callback for node 69c786e429015d0f959934f0 INFO: 172.20.0.1:54236 - "POST /node/69c786e429015d0f959934f0 HTTP/1.0" 202 Accepted 2026-03-30 07:44:42,821 - lava_callback - INFO - Starting async job submit for node 69c786e429015d0f959934f0 2026-03-30 07:44:42,822 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:44:42,836 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c786e429015d0f959934f0 HTTP/1.1" 200 None 2026-03-30 07:44:42,837 - lava_callback - DEBUG - Processing node 69c786e429015d0f959934f0: baseline-arm64-clabbe 2026-03-30 07:44:42,838 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c786e429015d0f959934f0 2026-03-30 07:44:42,838 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:44:42,852 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c786e429015d0f959934f0 HTTP/1.1" 200 None 2026-03-30 07:44:42,853 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:44:42,867 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 07:44:42,868 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:44:42,888 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c786e429015d0f959934f0 HTTP/1.1" 200 None 2026-03-30 07:44:42,889 - lava_callback - ERROR - Error processing callback for node 69c786e429015d0f959934f0: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54238 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38546 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54722 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42644 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42648 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47446 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47458 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58670 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44724 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39586 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:46:02,874 - lava_callback - INFO - Received callback for node 69c7873529015d0f959934fe INFO: 172.20.0.1:56392 - "POST /node/69c7873529015d0f959934fe HTTP/1.0" 202 Accepted 2026-03-30 07:46:02,880 - lava_callback - INFO - Starting async job submit for node 69c7873529015d0f959934fe 2026-03-30 07:46:02,881 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:46:02,923 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7873529015d0f959934fe HTTP/1.1" 200 None 2026-03-30 07:46:02,924 - lava_callback - DEBUG - Processing node 69c7873529015d0f959934fe: baseline-arm64-clabbe 2026-03-30 07:46:02,924 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7873529015d0f959934fe 2026-03-30 07:46:02,925 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:46:02,938 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7873529015d0f959934fe HTTP/1.1" 200 None 2026-03-30 07:46:02,940 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:46:02,953 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 07:46:02,954 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:46:02,998 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7873529015d0f959934fe HTTP/1.1" 200 None 2026-03-30 07:46:03,003 - lava_callback - ERROR - Error processing callback for node 69c7873529015d0f959934fe: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:56400 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51264 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51266 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47638 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47652 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46480 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33828 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52380 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59062 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34070 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34082 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:47:22,929 - lava_callback - INFO - Received callback for node 69c7878629015d0f95993506 INFO: 172.20.0.1:49214 - "POST /node/69c7878629015d0f95993506 HTTP/1.0" 202 Accepted 2026-03-30 07:47:22,934 - lava_callback - INFO - Starting async job submit for node 69c7878629015d0f95993506 2026-03-30 07:47:22,935 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:47:22,949 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7878629015d0f95993506 HTTP/1.1" 200 None 2026-03-30 07:47:22,950 - lava_callback - DEBUG - Processing node 69c7878629015d0f95993506: baseline-arm64-clabbe 2026-03-30 07:47:22,950 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7878629015d0f95993506 2026-03-30 07:47:22,951 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:47:22,968 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7878629015d0f95993506 HTTP/1.1" 200 None 2026-03-30 07:47:22,969 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:47:22,984 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c71e4e29015d0f9598f750 HTTP/1.1" 200 None 2026-03-30 07:47:22,985 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:47:23,006 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7878629015d0f95993506 HTTP/1.1" 200 None 2026-03-30 07:47:23,013 - lava_callback - ERROR - Error processing callback for node 69c7878629015d0f95993506: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49224 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49240 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58930 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36954 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38214 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43242 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52644 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52646 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50924 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50930 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59064 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:48:45,411 - lava_callback - INFO - Received callback for node 69c787cd29015d0f95993515 INFO: 172.20.0.1:54134 - "POST /node/69c787cd29015d0f95993515 HTTP/1.0" 202 Accepted 2026-03-30 07:48:45,416 - lava_callback - INFO - Starting async job submit for node 69c787cd29015d0f95993515 2026-03-30 07:48:45,417 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:48:45,431 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c787cd29015d0f95993515 HTTP/1.1" 200 None 2026-03-30 07:48:45,432 - lava_callback - DEBUG - Processing node 69c787cd29015d0f95993515: baseline-arm64-baylibre 2026-03-30 07:48:45,432 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c787cd29015d0f95993515 2026-03-30 07:48:45,433 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:48:45,456 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c787cd29015d0f95993515 HTTP/1.1" 200 None 2026-03-30 07:48:45,458 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:48:45,472 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 07:48:45,474 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:48:45,494 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c787cd29015d0f95993515 HTTP/1.1" 200 None 2026-03-30 07:48:45,501 - lava_callback - ERROR - Error processing callback for node 69c787cd29015d0f95993515: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54142 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35550 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48154 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60182 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60190 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40888 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40892 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42730 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51456 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43444 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:50:05,421 - lava_callback - INFO - Received callback for node 69c7882029015d0f95993522 INFO: 172.20.0.1:57342 - "POST /node/69c7882029015d0f95993522 HTTP/1.0" 202 Accepted 2026-03-30 07:50:05,427 - lava_callback - INFO - Starting async job submit for node 69c7882029015d0f95993522 2026-03-30 07:50:05,428 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:50:05,442 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7882029015d0f95993522 HTTP/1.1" 200 None 2026-03-30 07:50:05,443 - lava_callback - DEBUG - Processing node 69c7882029015d0f95993522: baseline-arm64-baylibre 2026-03-30 07:50:05,443 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7882029015d0f95993522 2026-03-30 07:50:05,444 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:50:05,456 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7882029015d0f95993522 HTTP/1.1" 200 None 2026-03-30 07:50:05,458 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:50:05,476 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 07:50:05,477 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:50:05,498 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7882029015d0f95993522 HTTP/1.1" 200 None 2026-03-30 07:50:05,503 - lava_callback - ERROR - Error processing callback for node 69c7882029015d0f95993522: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:57346 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46270 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46272 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37214 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37220 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47814 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37714 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44042 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34422 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57832 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:57838 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49710 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:51:25,451 - lava_callback - INFO - Received callback for node 69c7886d29015d0f95993527 INFO: 172.20.0.1:49716 - "POST /node/69c7886d29015d0f95993527 HTTP/1.0" 202 Accepted 2026-03-30 07:51:25,456 - lava_callback - INFO - Starting async job submit for node 69c7886d29015d0f95993527 2026-03-30 07:51:25,457 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:51:25,473 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7886d29015d0f95993527 HTTP/1.1" 200 None 2026-03-30 07:51:25,474 - lava_callback - DEBUG - Processing node 69c7886d29015d0f95993527: baseline-arm64-baylibre 2026-03-30 07:51:25,474 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7886d29015d0f95993527 2026-03-30 07:51:25,475 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:51:25,489 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7886d29015d0f95993527 HTTP/1.1" 200 None 2026-03-30 07:51:25,490 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:51:25,503 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 07:51:25,505 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:51:25,526 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7886d29015d0f95993527 HTTP/1.1" 200 None 2026-03-30 07:51:25,532 - lava_callback - ERROR - Error processing callback for node 69c7886d29015d0f95993527: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:49726 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52272 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58256 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34572 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57988 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33400 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33412 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46786 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46790 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35676 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:52:45,546 - lava_callback - INFO - Received callback for node 69c788c129015d0f95993533 INFO: 172.20.0.1:41672 - "POST /node/69c788c129015d0f95993533 HTTP/1.0" 202 Accepted 2026-03-30 07:52:45,551 - lava_callback - INFO - Starting async job submit for node 69c788c129015d0f95993533 2026-03-30 07:52:45,552 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:52:45,580 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c788c129015d0f95993533 HTTP/1.1" 200 None 2026-03-30 07:52:45,581 - lava_callback - DEBUG - Processing node 69c788c129015d0f95993533: baseline-arm64-baylibre 2026-03-30 07:52:45,581 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c788c129015d0f95993533 2026-03-30 07:52:45,582 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:52:45,600 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c788c129015d0f95993533 HTTP/1.1" 200 None 2026-03-30 07:52:45,602 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:52:45,616 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b4a590270ad7b6d699d HTTP/1.1" 200 None 2026-03-30 07:52:45,618 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:52:45,639 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c788c129015d0f95993533 HTTP/1.1" 200 None 2026-03-30 07:52:45,645 - lava_callback - ERROR - Error processing callback for node 69c788c129015d0f95993533: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:41684 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39916 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57498 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52100 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52116 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42226 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42242 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52362 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38598 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39132 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:54:05,647 - lava_callback - INFO - Received callback for node 69c7891129015d0f95993541 INFO: 172.20.0.1:48252 - "POST /node/69c7891129015d0f95993541 HTTP/1.0" 202 Accepted 2026-03-30 07:54:05,652 - lava_callback - INFO - Starting async job submit for node 69c7891129015d0f95993541 2026-03-30 07:54:05,653 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:54:05,668 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7891129015d0f95993541 HTTP/1.1" 200 None 2026-03-30 07:54:05,669 - lava_callback - DEBUG - Processing node 69c7891129015d0f95993541: baseline-arm64-baylibre 2026-03-30 07:54:05,669 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7891129015d0f95993541 2026-03-30 07:54:05,670 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:54:05,683 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7891129015d0f95993541 HTTP/1.1" 200 None 2026-03-30 07:54:05,685 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:54:05,704 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 07:54:05,705 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:54:05,726 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7891129015d0f95993541 HTTP/1.1" 200 None 2026-03-30 07:54:05,734 - lava_callback - ERROR - Error processing callback for node 69c7891129015d0f95993541: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48258 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39886 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39898 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49302 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49312 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50240 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42092 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33310 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41896 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60294 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:60304 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48868 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48874 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33254 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:55:43,264 - lava_callback - INFO - Received callback for node 69c7896b29015d0f9599354d INFO: 172.20.0.1:54430 - "POST /node/69c7896b29015d0f9599354d HTTP/1.0" 202 Accepted 2026-03-30 07:55:43,270 - lava_callback - INFO - Starting async job submit for node 69c7896b29015d0f9599354d 2026-03-30 07:55:43,271 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:55:43,285 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7896b29015d0f9599354d HTTP/1.1" 200 None 2026-03-30 07:55:43,286 - lava_callback - DEBUG - Processing node 69c7896b29015d0f9599354d: baseline-arm64-clabbe 2026-03-30 07:55:43,286 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c7896b29015d0f9599354d 2026-03-30 07:55:43,287 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:55:43,300 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c7896b29015d0f9599354d HTTP/1.1" 200 None 2026-03-30 07:55:43,301 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:55:43,314 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b4a590270ad7b6d699d HTTP/1.1" 200 None 2026-03-30 07:55:43,316 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:55:43,338 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c7896b29015d0f9599354d HTTP/1.1" 200 None 2026-03-30 07:55:43,344 - lava_callback - ERROR - Error processing callback for node 69c7896b29015d0f9599354d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54440 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54316 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51308 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39042 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:39052 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41580 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41590 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50734 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39374 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47916 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:57:03,305 - lava_callback - INFO - Received callback for node 69c789bb29015d0f95993555 INFO: 172.20.0.1:51250 - "POST /node/69c789bb29015d0f95993555 HTTP/1.0" 202 Accepted 2026-03-30 07:57:03,311 - lava_callback - INFO - Starting async job submit for node 69c789bb29015d0f95993555 2026-03-30 07:57:03,312 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:57:03,326 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c789bb29015d0f95993555 HTTP/1.1" 200 None 2026-03-30 07:57:03,327 - lava_callback - DEBUG - Processing node 69c789bb29015d0f95993555: baseline-arm64-clabbe 2026-03-30 07:57:03,327 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c789bb29015d0f95993555 2026-03-30 07:57:03,327 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:57:03,344 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c789bb29015d0f95993555 HTTP/1.1" 200 None 2026-03-30 07:57:03,345 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:57:03,360 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b4a590270ad7b6d699d HTTP/1.1" 200 None 2026-03-30 07:57:03,361 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:57:03,381 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c789bb29015d0f95993555 HTTP/1.1" 200 None 2026-03-30 07:57:03,387 - lava_callback - ERROR - Error processing callback for node 69c789bb29015d0f95993555: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:51252 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:49272 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:49278 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42886 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42900 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44258 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35876 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44444 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:58:05,856 - lava_callback - INFO - Received callback for node 69c78a0529015d0f95993560 INFO: 172.20.0.1:60460 - "POST /node/69c78a0529015d0f95993560 HTTP/1.0" 202 Accepted 2026-03-30 07:58:05,861 - lava_callback - INFO - Starting async job submit for node 69c78a0529015d0f95993560 2026-03-30 07:58:05,862 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:58:05,877 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78a0529015d0f95993560 HTTP/1.1" 200 None 2026-03-30 07:58:05,878 - lava_callback - DEBUG - Processing node 69c78a0529015d0f95993560: baseline-arm64-baylibre 2026-03-30 07:58:05,878 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c78a0529015d0f95993560 2026-03-30 07:58:05,879 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:58:05,893 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78a0529015d0f95993560 HTTP/1.1" 200 None 2026-03-30 07:58:05,894 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:58:05,907 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 07:58:05,908 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:58:05,934 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c78a0529015d0f95993560 HTTP/1.1" 200 None 2026-03-30 07:58:05,935 - lava_callback - ERROR - Error processing callback for node 69c78a0529015d0f95993560: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:60466 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34722 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:34738 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36400 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36412 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37852 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55504 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38262 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56156 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51134 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51142 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32924 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:32940 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36622 - "GET / HTTP/1.0" 200 OK 2026-03-30 07:59:43,445 - lava_callback - INFO - Received callback for node 69c78a5c29015d0f9599356a INFO: 172.20.0.1:36674 - "POST /node/69c78a5c29015d0f9599356a HTTP/1.0" 202 Accepted 2026-03-30 07:59:43,451 - lava_callback - INFO - Starting async job submit for node 69c78a5c29015d0f9599356a 2026-03-30 07:59:43,452 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:59:43,466 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78a5c29015d0f9599356a HTTP/1.1" 200 None 2026-03-30 07:59:43,467 - lava_callback - DEBUG - Processing node 69c78a5c29015d0f9599356a: baseline-arm64-clabbe 2026-03-30 07:59:43,468 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c78a5c29015d0f9599356a 2026-03-30 07:59:43,468 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:59:43,481 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78a5c29015d0f9599356a HTTP/1.1" 200 None 2026-03-30 07:59:43,483 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:59:43,496 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b4a590270ad7b6d699d HTTP/1.1" 200 None 2026-03-30 07:59:43,497 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 07:59:43,518 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c78a5c29015d0f9599356a HTTP/1.1" 200 None 2026-03-30 07:59:43,519 - lava_callback - ERROR - Error processing callback for node 69c78a5c29015d0f9599356a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:36680 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50052 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53948 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35880 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35896 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35908 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55670 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55682 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55688 - "GET /favicon.ico HTTP/1.0" 404 Not Found INFO: 172.20.0.1:38198 - "GET / HTTP/1.0" 200 OK 2026-03-30 08:00:45,976 - lava_callback - INFO - Received callback for node 69c78aa729015d0f95993570 INFO: 172.20.0.1:44136 - "POST /node/69c78aa729015d0f95993570 HTTP/1.0" 202 Accepted 2026-03-30 08:00:45,982 - lava_callback - INFO - Starting async job submit for node 69c78aa729015d0f95993570 2026-03-30 08:00:45,983 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:00:45,998 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78aa729015d0f95993570 HTTP/1.1" 200 None 2026-03-30 08:00:46,000 - lava_callback - DEBUG - Processing node 69c78aa729015d0f95993570: baseline-arm64-baylibre 2026-03-30 08:00:46,000 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c78aa729015d0f95993570 2026-03-30 08:00:46,001 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:00:46,015 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78aa729015d0f95993570 HTTP/1.1" 200 None 2026-03-30 08:00:46,016 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:00:46,030 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 08:00:46,031 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:00:46,052 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c78aa729015d0f95993570 HTTP/1.1" 200 None 2026-03-30 08:00:46,053 - lava_callback - ERROR - Error processing callback for node 69c78aa729015d0f95993570: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:44140 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57896 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44370 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48008 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48016 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33900 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33910 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42086 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35100 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:54148 - "GET / HTTP/1.0" 200 OK 2026-03-30 08:02:06,061 - lava_callback - INFO - Received callback for node 69c78af829015d0f95993585 INFO: 172.20.0.1:54788 - "POST /node/69c78af829015d0f95993585 HTTP/1.0" 202 Accepted 2026-03-30 08:02:06,066 - lava_callback - INFO - Starting async job submit for node 69c78af829015d0f95993585 2026-03-30 08:02:06,067 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:02:06,087 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78af829015d0f95993585 HTTP/1.1" 200 None 2026-03-30 08:02:06,088 - lava_callback - DEBUG - Processing node 69c78af829015d0f95993585: baseline-arm64-baylibre 2026-03-30 08:02:06,088 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c78af829015d0f95993585 2026-03-30 08:02:06,089 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:02:06,102 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78af829015d0f95993585 HTTP/1.1" 200 None 2026-03-30 08:02:06,104 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:02:06,138 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b4a590270ad7b6d699d HTTP/1.1" 200 None 2026-03-30 08:02:06,139 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:02:06,160 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c78af829015d0f95993585 HTTP/1.1" 200 None 2026-03-30 08:02:06,161 - lava_callback - ERROR - Error processing callback for node 69c78af829015d0f95993585: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:54798 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52568 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:52572 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38734 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38744 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51644 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41540 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:44712 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38184 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48748 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:48752 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47312 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47322 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53824 - "GET / HTTP/1.0" 200 OK 2026-03-30 08:03:43,614 - lava_callback - INFO - Received callback for node 69c78b4f29015d0f95993593 INFO: 172.20.0.1:46070 - "POST /node/69c78b4f29015d0f95993593 HTTP/1.0" 202 Accepted 2026-03-30 08:03:43,620 - lava_callback - INFO - Starting async job submit for node 69c78b4f29015d0f95993593 2026-03-30 08:03:43,621 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:03:43,634 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78b4f29015d0f95993593 HTTP/1.1" 200 None 2026-03-30 08:03:43,635 - lava_callback - DEBUG - Processing node 69c78b4f29015d0f95993593: baseline-arm64-clabbe 2026-03-30 08:03:43,635 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c78b4f29015d0f95993593 2026-03-30 08:03:43,635 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:03:43,648 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78b4f29015d0f95993593 HTTP/1.1" 200 None 2026-03-30 08:03:43,650 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:03:43,662 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b4a590270ad7b6d699d HTTP/1.1" 200 None 2026-03-30 08:03:43,664 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:03:43,684 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c78b4f29015d0f95993593 HTTP/1.1" 200 None 2026-03-30 08:03:43,689 - lava_callback - ERROR - Error processing callback for node 69c78b4f29015d0f95993593: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:46076 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46090 - "GET /wiki HTTP/1.0" 404 Not Found INFO: 172.20.0.1:46464 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36602 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46278 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:46282 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47286 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47298 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59956 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34754 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:55946 - "GET / HTTP/1.0" 200 OK 2026-03-30 08:05:03,769 - lava_callback - INFO - Received callback for node 69c78b9f29015d0f9599359c INFO: 172.20.0.1:48778 - "POST /node/69c78b9f29015d0f9599359c HTTP/1.0" 202 Accepted 2026-03-30 08:05:03,775 - lava_callback - INFO - Starting async job submit for node 69c78b9f29015d0f9599359c 2026-03-30 08:05:03,776 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:05:03,790 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78b9f29015d0f9599359c HTTP/1.1" 200 None 2026-03-30 08:05:03,791 - lava_callback - DEBUG - Processing node 69c78b9f29015d0f9599359c: baseline-arm64-clabbe 2026-03-30 08:05:03,791 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c78b9f29015d0f9599359c 2026-03-30 08:05:03,791 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:05:03,805 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78b9f29015d0f9599359c HTTP/1.1" 200 None 2026-03-30 08:05:03,806 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:05:03,819 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b4a590270ad7b6d699d HTTP/1.1" 200 None 2026-03-30 08:05:03,820 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:05:03,840 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c78b9f29015d0f9599359c HTTP/1.1" 200 None 2026-03-30 08:05:03,846 - lava_callback - ERROR - Error processing callback for node 69c78b9f29015d0f9599359c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:48786 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43168 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:43176 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39698 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39712 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43140 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41098 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43928 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39422 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42198 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:42206 - "GET / HTTP/1.0" 200 OK 2026-03-30 08:06:23,718 - lava_callback - INFO - Received callback for node 69c78bf129015d0f959935a7 INFO: 172.20.0.1:47466 - "POST /node/69c78bf129015d0f959935a7 HTTP/1.0" 202 Accepted 2026-03-30 08:06:23,723 - lava_callback - INFO - Starting async job submit for node 69c78bf129015d0f959935a7 2026-03-30 08:06:23,724 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:06:23,757 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78bf129015d0f959935a7 HTTP/1.1" 200 None 2026-03-30 08:06:23,758 - lava_callback - DEBUG - Processing node 69c78bf129015d0f959935a7: baseline-arm64-clabbe 2026-03-30 08:06:23,758 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c78bf129015d0f959935a7 2026-03-30 08:06:23,759 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:06:23,771 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78bf129015d0f959935a7 HTTP/1.1" 200 None 2026-03-30 08:06:23,773 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:06:23,785 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b4a590270ad7b6d699d HTTP/1.1" 200 None 2026-03-30 08:06:23,787 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:06:23,808 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c78bf129015d0f959935a7 HTTP/1.1" 200 None 2026-03-30 08:06:23,810 - lava_callback - ERROR - Error processing callback for node 69c78bf129015d0f959935a7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:47472 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:47482 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45726 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:57788 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50216 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42752 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59696 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:59700 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33894 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:33910 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60666 - "GET / HTTP/1.0" 200 OK 2026-03-30 08:07:43,745 - lava_callback - INFO - Received callback for node 69c78c4129015d0f959935b1 INFO: 172.20.0.1:58840 - "POST /node/69c78c4129015d0f959935b1 HTTP/1.0" 202 Accepted 2026-03-30 08:07:43,750 - lava_callback - INFO - Starting async job submit for node 69c78c4129015d0f959935b1 2026-03-30 08:07:43,751 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:07:43,777 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78c4129015d0f959935b1 HTTP/1.1" 200 None 2026-03-30 08:07:43,778 - lava_callback - DEBUG - Processing node 69c78c4129015d0f959935b1: baseline-arm64-clabbe 2026-03-30 08:07:43,778 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c78c4129015d0f959935b1 2026-03-30 08:07:43,779 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:07:43,792 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78c4129015d0f959935b1 HTTP/1.1" 200 None 2026-03-30 08:07:43,793 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:07:43,807 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b4a590270ad7b6d699d HTTP/1.1" 200 None 2026-03-30 08:07:43,808 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:07:43,837 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c78c4129015d0f959935b1 HTTP/1.1" 200 None 2026-03-30 08:07:43,842 - lava_callback - ERROR - Error processing callback for node 69c78c4129015d0f959935b1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:58844 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:39394 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51826 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37524 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37534 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58588 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58604 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34306 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:41412 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37518 - "GET / HTTP/1.0" 200 OK 2026-03-30 08:09:03,826 - lava_callback - INFO - Received callback for node 69c78c9129015d0f959935ba INFO: 172.20.0.1:33404 - "POST /node/69c78c9129015d0f959935ba HTTP/1.0" 202 Accepted 2026-03-30 08:09:03,832 - lava_callback - INFO - Starting async job submit for node 69c78c9129015d0f959935ba 2026-03-30 08:09:03,833 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:09:03,868 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78c9129015d0f959935ba HTTP/1.1" 200 None 2026-03-30 08:09:03,869 - lava_callback - DEBUG - Processing node 69c78c9129015d0f959935ba: baseline-arm64-clabbe 2026-03-30 08:09:03,869 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c78c9129015d0f959935ba 2026-03-30 08:09:03,870 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:09:03,883 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78c9129015d0f959935ba HTTP/1.1" 200 None 2026-03-30 08:09:03,885 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:09:03,897 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b4a590270ad7b6d699d HTTP/1.1" 200 None 2026-03-30 08:09:03,899 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:09:03,920 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c78c9129015d0f959935ba HTTP/1.1" 200 None 2026-03-30 08:09:03,921 - lava_callback - ERROR - Error processing callback for node 69c78c9129015d0f959935ba: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:33418 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35762 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:35774 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42758 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:42774 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:52736 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:48714 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40214 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59288 - "GET / HTTP/1.0" 200 OK 2026-03-30 08:10:06,530 - lava_callback - INFO - Received callback for node 69c78cd729015d0f959935c7 INFO: 172.20.0.1:59296 - "POST /node/69c78cd729015d0f959935c7 HTTP/1.0" 202 Accepted 2026-03-30 08:10:06,536 - lava_callback - INFO - Starting async job submit for node 69c78cd729015d0f959935c7 2026-03-30 08:10:06,537 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:10:06,552 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78cd729015d0f959935c7 HTTP/1.1" 200 None 2026-03-30 08:10:06,553 - lava_callback - DEBUG - Processing node 69c78cd729015d0f959935c7: baseline-arm64-baylibre 2026-03-30 08:10:06,553 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c78cd729015d0f959935c7 2026-03-30 08:10:06,554 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:10:06,589 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78cd729015d0f959935c7 HTTP/1.1" 200 None 2026-03-30 08:10:06,591 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:10:06,606 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 08:10:06,608 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:10:06,630 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c78cd729015d0f959935c7 HTTP/1.1" 200 None 2026-03-30 08:10:06,632 - lava_callback - ERROR - Error processing callback for node 69c78cd729015d0f959935c7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:33376 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:33386 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58990 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58998 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:50474 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:36378 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:35848 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56196 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:45032 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:45044 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46798 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46806 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:46122 - "GET / HTTP/1.0" 200 OK 2026-03-30 08:11:44,015 - lava_callback - INFO - Received callback for node 69c78d3029015d0f959935d3 INFO: 172.20.0.1:38550 - "POST /node/69c78d3029015d0f959935d3 HTTP/1.0" 202 Accepted 2026-03-30 08:11:44,020 - lava_callback - INFO - Starting async job submit for node 69c78d3029015d0f959935d3 2026-03-30 08:11:44,021 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:11:44,036 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78d3029015d0f959935d3 HTTP/1.1" 200 None 2026-03-30 08:11:44,037 - lava_callback - DEBUG - Processing node 69c78d3029015d0f959935d3: baseline-arm64-clabbe 2026-03-30 08:11:44,037 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c78d3029015d0f959935d3 2026-03-30 08:11:44,038 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:11:44,096 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78d3029015d0f959935d3 HTTP/1.1" 200 None 2026-03-30 08:11:44,099 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:11:44,119 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c47b4a590270ad7b6d699d HTTP/1.1" 200 None 2026-03-30 08:11:44,121 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:11:44,167 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c78d3029015d0f959935d3 HTTP/1.1" 200 None 2026-03-30 08:11:44,169 - lava_callback - ERROR - Error processing callback for node 69c78d3029015d0f959935d3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:38552 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:34762 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:40776 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:51582 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:51586 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56476 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:56480 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38670 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:43648 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:53900 - "GET / HTTP/1.0" 200 OK 2026-03-30 08:13:03,997 - lava_callback - INFO - Received callback for node 69c78d8229015d0f959935d9 INFO: 172.20.0.1:37946 - "POST /node/69c78d8229015d0f959935d9 HTTP/1.0" 202 Accepted 2026-03-30 08:13:04,002 - lava_callback - INFO - Starting async job submit for node 69c78d8229015d0f959935d9 2026-03-30 08:13:04,003 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:13:04,023 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78d8229015d0f959935d9 HTTP/1.1" 200 None 2026-03-30 08:13:04,024 - lava_callback - DEBUG - Processing node 69c78d8229015d0f959935d9: baseline-arm64-clabbe 2026-03-30 08:13:04,024 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69c78d8229015d0f959935d9 2026-03-30 08:13:04,024 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:13:04,037 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c78d8229015d0f959935d9 HTTP/1.1" 200 None 2026-03-30 08:13:04,039 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:13:04,062 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69c1d8400efaf21d93103552 HTTP/1.1" 200 None 2026-03-30 08:13:04,065 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-30 08:13:04,090 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69c78d8229015d0f959935d9 HTTP/1.1" 200 None 2026-03-30 08:13:04,092 - lava_callback - ERROR - Error processing callback for node 69c78d8229015d0f959935d9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.20.0.1:37952 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:37668 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.20.0.1:37672 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60840 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:60848 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:58818 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:59148 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38240 - "GET / HTTP/1.0" 200 OK INFO: 172.20.0.1:38756 - "GET / HTTP/1.0" 200 OK INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [1]