INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) INFO: 172.21.0.1:58106 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47860 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50988 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60016 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:60028 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60038 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55156 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35754 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51440 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56374 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40040 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42830 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:42834 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42836 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49418 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36834 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55706 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40192 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36980 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37306 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:37320 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37332 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39866 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48492 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33352 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50510 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47912 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56008 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56014 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56024 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53522 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35248 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46758 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42852 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57794 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59780 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59786 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59792 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59656 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39016 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40662 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37540 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53700 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45380 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:45382 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45388 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55806 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49776 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48412 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33732 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58414 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45002 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:45004 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45020 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36156 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60592 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51230 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56328 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46464 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49092 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:49094 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49102 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47986 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53504 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42800 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39498 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56702 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56892 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56900 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56910 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44824 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40960 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34600 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34384 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49992 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59764 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59776 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59784 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53796 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47638 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42768 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35458 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33928 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47436 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47450 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47462 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45280 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33062 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45718 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36796 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39008 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40804 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:40814 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35760 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35764 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38134 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51806 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44456 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56400 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43602 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:43614 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44674 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44684 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51346 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51342 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43684 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59816 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50092 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:50100 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36110 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36114 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48058 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59578 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45742 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37644 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56216 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56224 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49444 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49446 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38762 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33444 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44064 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34830 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56398 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56402 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50514 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50518 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50816 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46004 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59336 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40428 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41088 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:41096 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36916 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36918 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48252 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49016 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46534 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51148 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48758 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48766 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46156 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46172 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37604 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38772 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40076 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58552 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48964 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48978 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43824 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43838 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38222 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45704 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56326 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37856 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33986 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34000 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54456 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54462 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33352 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54698 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47608 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32916 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47254 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47258 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50490 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50496 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54276 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39918 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38458 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42120 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57816 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:57820 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42682 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42696 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37422 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54662 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38614 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36500 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58736 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58752 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35860 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35870 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55214 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43296 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51960 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56716 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40892 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:40902 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56986 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56994 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33000 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37250 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58164 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41584 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:38:13,801 - lava_callback - INFO - Received callback for node 69aa21ed8c4d487a0e7512f0 2026-03-08 00:38:13,808 - lava_callback - INFO - Starting async job submit for node 69aa21ed8c4d487a0e7512f0 INFO: 172.21.0.1:32850 - "POST /node/69aa21ed8c4d487a0e7512f0 HTTP/1.0" 202 Accepted 2026-03-08 00:38:13,810 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:38:13,831 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa21ed8c4d487a0e7512f0 HTTP/1.1" 200 None 2026-03-08 00:38:13,832 - lava_callback - DEBUG - Processing node 69aa21ed8c4d487a0e7512f0: baseline-arm-clabbe 2026-03-08 00:38:13,833 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa21ed8c4d487a0e7512f0 2026-03-08 00:38:13,833 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:38:13,855 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa21ed8c4d487a0e7512f0 HTTP/1.1" 200 None 2026-03-08 00:38:13,857 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:38:13,882 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa1d368c4d487a0e751145 HTTP/1.1" 200 None 2026-03-08 00:38:13,883 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:38:13,909 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa21ed8c4d487a0e7512f0 HTTP/1.1" 200 None 2026-03-08 00:38:13,911 - lava_callback - ERROR - Error processing callback for node 69aa21ed8c4d487a0e7512f0: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-08 00:38:14,481 - lava_callback - INFO - Received callback for node 69aa21ee8c4d487a0e7512f3 INFO: 172.21.0.1:32856 - "POST /node/69aa21ee8c4d487a0e7512f3 HTTP/1.0" 202 Accepted 2026-03-08 00:38:14,488 - lava_callback - INFO - Starting async job submit for node 69aa21ee8c4d487a0e7512f3 2026-03-08 00:38:14,489 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:38:14,505 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa21ee8c4d487a0e7512f3 HTTP/1.1" 200 None 2026-03-08 00:38:14,506 - lava_callback - DEBUG - Processing node 69aa21ee8c4d487a0e7512f3: baseline-arm-clabbe 2026-03-08 00:38:14,507 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa21ee8c4d487a0e7512f3 2026-03-08 00:38:14,507 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:38:14,525 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa21ee8c4d487a0e7512f3 HTTP/1.1" 200 None 2026-03-08 00:38:14,527 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:38:14,542 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa1d368c4d487a0e751145 HTTP/1.1" 200 None 2026-03-08 00:38:14,543 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:38:14,563 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa21ee8c4d487a0e7512f3 HTTP/1.1" 200 None 2026-03-08 00:38:14,564 - lava_callback - ERROR - Error processing callback for node 69aa21ee8c4d487a0e7512f3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-08 00:38:15,132 - lava_callback - INFO - Received callback for node 69aa21f28c4d487a0e7512ff INFO: 172.21.0.1:32868 - "POST /node/69aa21f28c4d487a0e7512ff HTTP/1.0" 202 Accepted 2026-03-08 00:38:15,137 - lava_callback - INFO - Starting async job submit for node 69aa21f28c4d487a0e7512ff 2026-03-08 00:38:15,138 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:38:15,154 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa21f28c4d487a0e7512ff HTTP/1.1" 200 None 2026-03-08 00:38:15,155 - lava_callback - DEBUG - Processing node 69aa21f28c4d487a0e7512ff: baseline-arm-clabbe 2026-03-08 00:38:15,155 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa21f28c4d487a0e7512ff 2026-03-08 00:38:15,156 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.21.0.1:32880 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 00:38:15,175 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa21f28c4d487a0e7512ff HTTP/1.1" 200 None 2026-03-08 00:38:15,177 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:38:15,190 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa1d368c4d487a0e751145 HTTP/1.1" 200 None 2026-03-08 00:38:15,191 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:38:15,211 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa21f28c4d487a0e7512ff HTTP/1.1" 200 None 2026-03-08 00:38:15,213 - lava_callback - ERROR - Error processing callback for node 69aa21f28c4d487a0e7512ff: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:32896 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51688 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51694 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42054 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37850 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37958 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56852 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58252 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58254 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59956 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59962 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42282 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40592 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53172 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33140 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45866 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:45882 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36052 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36064 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:40:33,207 - lava_callback - INFO - Received callback for node 69acc500bcd207e915d76cde INFO: 172.21.0.1:52066 - "POST /node/69acc500bcd207e915d76cde HTTP/1.0" 202 Accepted 2026-03-08 00:40:33,213 - lava_callback - INFO - Starting async job submit for node 69acc500bcd207e915d76cde 2026-03-08 00:40:33,252 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:40:33,266 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc500bcd207e915d76cde HTTP/1.1" 200 None 2026-03-08 00:40:33,267 - lava_callback - DEBUG - Processing node 69acc500bcd207e915d76cde: baseline-arm 2026-03-08 00:40:33,453 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:40:33,683 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:40:33,685 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69acc500bcd207e915d76cde/log.txt.gz 2026-03-08 00:40:33,795 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:40:33,963 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:40:33,965 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69acc500bcd207e915d76cde/lava-logs.yaml 2026-03-08 00:40:33,970 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:40:34,107 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:40:34,109 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69acc500bcd207e915d76cde/lava_callback.json.gz 2026-03-08 00:40:34,135 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:40:34,155 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc500bcd207e915d76cde HTTP/1.1" 200 None 2026-03-08 00:40:34,157 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:40:34,170 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc012bcd207e915d76b9c HTTP/1.1" 200 None 2026-03-08 00:40:34,171 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:40:34,203 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc500bcd207e915d76cde HTTP/1.1" 200 None 2026-03-08 00:40:34,205 - lava_callback - INFO - Completed processing callback for node 69acc500bcd207e915d76cde INFO: 172.21.0.1:52070 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:40:43,912 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:40:43,931 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:37594 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33770 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45036 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37522 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:37526 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50776 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50780 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38690 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47922 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48786 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44156 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53928 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:53936 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39522 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39532 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42548 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44196 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46382 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38280 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35924 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:35936 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45102 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45114 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44540 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39586 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36254 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33056 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:44:11,880 - lava_callback - INFO - Received callback for node 69aa23538c4d487a0e75151d INFO: 172.21.0.1:45226 - "POST /node/69aa23538c4d487a0e75151d HTTP/1.0" 202 Accepted 2026-03-08 00:44:11,885 - lava_callback - INFO - Starting async job submit for node 69aa23538c4d487a0e75151d 2026-03-08 00:44:11,886 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:44:11,912 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa23538c4d487a0e75151d HTTP/1.1" 200 None 2026-03-08 00:44:11,913 - lava_callback - DEBUG - Processing node 69aa23538c4d487a0e75151d: baseline-arm-baylibre 2026-03-08 00:44:11,913 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa23538c4d487a0e75151d 2026-03-08 00:44:11,914 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:44:11,928 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa23538c4d487a0e75151d HTTP/1.1" 200 None 2026-03-08 00:44:11,929 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:44:11,943 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a77a57a255cc0324eb010a HTTP/1.1" 200 None 2026-03-08 00:44:11,944 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:44:11,965 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa23538c4d487a0e75151d HTTP/1.1" 200 None 2026-03-08 00:44:11,972 - lava_callback - ERROR - Error processing callback for node 69aa23538c4d487a0e75151d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:45240 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:45248 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48198 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48204 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57590 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36280 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:44:51,771 - lava_callback - INFO - Received callback for node 69aa23768c4d487a0e751537 INFO: 172.21.0.1:48346 - "POST /node/69aa23768c4d487a0e751537 HTTP/1.0" 202 Accepted 2026-03-08 00:44:51,776 - lava_callback - INFO - Starting async job submit for node 69aa23768c4d487a0e751537 2026-03-08 00:44:51,777 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:44:51,792 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa23768c4d487a0e751537 HTTP/1.1" 200 None 2026-03-08 00:44:51,793 - lava_callback - DEBUG - Processing node 69aa23768c4d487a0e751537: baseline-arm-baylibre 2026-03-08 00:44:51,794 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa23768c4d487a0e751537 2026-03-08 00:44:51,794 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:44:51,808 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa23768c4d487a0e751537 HTTP/1.1" 200 None 2026-03-08 00:44:51,809 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:44:51,822 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a77a57a255cc0324eb010a HTTP/1.1" 200 None 2026-03-08 00:44:51,823 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:44:51,844 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa23768c4d487a0e751537 HTTP/1.1" 200 None 2026-03-08 00:44:51,845 - lava_callback - ERROR - Error processing callback for node 69aa23768c4d487a0e751537: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:48356 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59018 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47910 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47926 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33708 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33712 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:45:31,825 - lava_callback - INFO - Received callback for node 69aa239b8c4d487a0e751553 INFO: 172.21.0.1:33686 - "POST /node/69aa239b8c4d487a0e751553 HTTP/1.0" 202 Accepted 2026-03-08 00:45:31,830 - lava_callback - INFO - Starting async job submit for node 69aa239b8c4d487a0e751553 2026-03-08 00:45:31,831 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:45:31,846 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa239b8c4d487a0e751553 HTTP/1.1" 200 None 2026-03-08 00:45:31,847 - lava_callback - DEBUG - Processing node 69aa239b8c4d487a0e751553: baseline-arm-baylibre 2026-03-08 00:45:31,847 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa239b8c4d487a0e751553 2026-03-08 00:45:31,848 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:45:31,860 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa239b8c4d487a0e751553 HTTP/1.1" 200 None 2026-03-08 00:45:31,862 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:45:31,878 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a77a57a255cc0324eb010a HTTP/1.1" 200 None 2026-03-08 00:45:31,879 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:45:31,899 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa239b8c4d487a0e751553 HTTP/1.1" 200 None 2026-03-08 00:45:31,904 - lava_callback - ERROR - Error processing callback for node 69aa239b8c4d487a0e751553: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:33696 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57220 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47768 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34126 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:46:11,842 - lava_callback - INFO - Received callback for node 69aa23c08c4d487a0e75156c INFO: 172.21.0.1:55330 - "POST /node/69aa23c08c4d487a0e75156c HTTP/1.0" 202 Accepted 2026-03-08 00:46:11,848 - lava_callback - INFO - Starting async job submit for node 69aa23c08c4d487a0e75156c 2026-03-08 00:46:11,849 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:46:11,863 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa23c08c4d487a0e75156c HTTP/1.1" 200 None 2026-03-08 00:46:11,864 - lava_callback - DEBUG - Processing node 69aa23c08c4d487a0e75156c: baseline-arm-baylibre 2026-03-08 00:46:11,864 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa23c08c4d487a0e75156c 2026-03-08 00:46:11,865 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:46:11,883 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa23c08c4d487a0e75156c HTTP/1.1" 200 None 2026-03-08 00:46:11,884 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:46:11,897 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a77a57a255cc0324eb010a HTTP/1.1" 200 None 2026-03-08 00:46:11,899 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:46:11,918 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa23c08c4d487a0e75156c HTTP/1.1" 200 None 2026-03-08 00:46:11,924 - lava_callback - ERROR - Error processing callback for node 69aa23c08c4d487a0e75156c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:55342 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55344 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42906 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42908 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:46:31,836 - lava_callback - INFO - Received callback for node 69aa23e58c4d487a0e75157b INFO: 172.21.0.1:53090 - "POST /node/69aa23e58c4d487a0e75157b HTTP/1.0" 202 Accepted 2026-03-08 00:46:31,842 - lava_callback - INFO - Starting async job submit for node 69aa23e58c4d487a0e75157b 2026-03-08 00:46:31,843 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:46:31,858 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa23e58c4d487a0e75157b HTTP/1.1" 200 None 2026-03-08 00:46:31,859 - lava_callback - DEBUG - Processing node 69aa23e58c4d487a0e75157b: baseline-arm-baylibre 2026-03-08 00:46:31,859 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa23e58c4d487a0e75157b 2026-03-08 00:46:31,860 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:46:31,874 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa23e58c4d487a0e75157b HTTP/1.1" 200 None 2026-03-08 00:46:31,875 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:46:31,888 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a77a57a255cc0324eb010a HTTP/1.1" 200 None 2026-03-08 00:46:31,889 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:46:31,911 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa23e58c4d487a0e75157b HTTP/1.1" 200 None 2026-03-08 00:46:31,912 - lava_callback - ERROR - Error processing callback for node 69aa23e58c4d487a0e75157b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:53100 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54658 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46180 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50742 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:47:11,947 - lava_callback - INFO - Received callback for node 69aa240b8c4d487a0e7515a1 INFO: 172.21.0.1:38486 - "POST /node/69aa240b8c4d487a0e7515a1 HTTP/1.0" 202 Accepted 2026-03-08 00:47:11,952 - lava_callback - INFO - Starting async job submit for node 69aa240b8c4d487a0e7515a1 2026-03-08 00:47:11,953 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:47:11,967 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa240b8c4d487a0e7515a1 HTTP/1.1" 200 None 2026-03-08 00:47:11,968 - lava_callback - DEBUG - Processing node 69aa240b8c4d487a0e7515a1: baseline-arm-baylibre 2026-03-08 00:47:11,968 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa240b8c4d487a0e7515a1 2026-03-08 00:47:11,969 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:47:11,983 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa240b8c4d487a0e7515a1 HTTP/1.1" 200 None 2026-03-08 00:47:11,984 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:47:11,997 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a77a57a255cc0324eb010a HTTP/1.1" 200 None 2026-03-08 00:47:11,999 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:47:12,033 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa240b8c4d487a0e7515a1 HTTP/1.1" 200 None 2026-03-08 00:47:12,039 - lava_callback - ERROR - Error processing callback for node 69aa240b8c4d487a0e7515a1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:38488 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:38502 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33074 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33084 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51874 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:47:45,815 - lava_callback - INFO - Received callback for node 69acc714bcd207e915d76dc5 INFO: 172.21.0.1:44158 - "POST /node/69acc714bcd207e915d76dc5 HTTP/1.0" 202 Accepted 2026-03-08 00:47:45,821 - lava_callback - INFO - Starting async job submit for node 69acc714bcd207e915d76dc5 2026-03-08 00:47:45,839 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:47:45,853 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc714bcd207e915d76dc5 HTTP/1.1" 200 None 2026-03-08 00:47:45,854 - lava_callback - DEBUG - Processing node 69acc714bcd207e915d76dc5: baseline-x86 2026-03-08 00:47:46,038 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:47:46,187 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:47:46,189 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69acc714bcd207e915d76dc5/log.txt.gz 2026-03-08 00:47:46,301 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:44164 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:47:46,533 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:47:46,535 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69acc714bcd207e915d76dc5/lava-logs.yaml 2026-03-08 00:47:46,540 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:47:46,731 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:47:46,733 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69acc714bcd207e915d76dc5/lava_callback.json.gz 2026-03-08 00:47:46,751 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:47:46,766 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc714bcd207e915d76dc5 HTTP/1.1" 200 None 2026-03-08 00:47:46,767 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:47:46,781 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc037bcd207e915d76bc8 HTTP/1.1" 200 None 2026-03-08 00:47:46,782 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:47:46,806 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc714bcd207e915d76dc5 HTTP/1.1" 200 None 2026-03-08 00:47:46,808 - lava_callback - INFO - Completed processing callback for node 69acc714bcd207e915d76dc5 2026-03-08 00:47:50,597 - lava_callback - INFO - Received callback for node 69acc716bcd207e915d76dcf INFO: 172.21.0.1:52352 - "POST /node/69acc716bcd207e915d76dcf HTTP/1.0" 202 Accepted 2026-03-08 00:47:50,603 - lava_callback - INFO - Starting async job submit for node 69acc716bcd207e915d76dcf 2026-03-08 00:47:50,620 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:47:50,634 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc716bcd207e915d76dcf HTTP/1.1" 200 None 2026-03-08 00:47:50,635 - lava_callback - DEBUG - Processing node 69acc716bcd207e915d76dcf: baseline-x86 2026-03-08 00:47:50,803 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:47:50,933 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:47:50,935 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69acc716bcd207e915d76dcf/log.txt.gz 2026-03-08 00:47:51,037 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:47:51,228 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:47:51,230 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69acc716bcd207e915d76dcf/lava-logs.yaml 2026-03-08 00:47:51,235 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:47:51,399 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:47:51,401 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69acc716bcd207e915d76dcf/lava_callback.json.gz 2026-03-08 00:47:51,432 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:47:51,453 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc716bcd207e915d76dcf HTTP/1.1" 200 None 2026-03-08 00:47:51,455 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:47:51,478 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc037bcd207e915d76bc8 HTTP/1.1" 200 None 2026-03-08 00:47:51,480 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:47:51,523 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc716bcd207e915d76dcf HTTP/1.1" 200 None 2026-03-08 00:47:51,525 - lava_callback - INFO - Completed processing callback for node 69acc716bcd207e915d76dcf INFO: 172.21.0.1:52368 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46648 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:48:13,935 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:48:13,963 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:55324 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55336 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53258 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53260 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48342 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35744 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39662 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:49:03,742 - lava_callback - INFO - Received callback for node 69acc715bcd207e915d76dc8 INFO: 172.21.0.1:53464 - "POST /node/69acc715bcd207e915d76dc8 HTTP/1.0" 202 Accepted 2026-03-08 00:49:03,752 - lava_callback - INFO - Starting async job submit for node 69acc715bcd207e915d76dc8 2026-03-08 00:49:03,773 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:49:03,788 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc715bcd207e915d76dc8 HTTP/1.1" 200 None 2026-03-08 00:49:03,789 - lava_callback - DEBUG - Processing node 69acc715bcd207e915d76dc8: baseline-x86 2026-03-08 00:49:04,198 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:49:04,858 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:49:04,861 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69acc715bcd207e915d76dc8/log.txt.gz 2026-03-08 00:49:05,146 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:49:05,463 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:49:05,466 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69acc715bcd207e915d76dc8/lava-logs.yaml 2026-03-08 00:49:05,490 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:49:06,166 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:49:06,168 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69acc715bcd207e915d76dc8/lava_callback.json.gz 2026-03-08 00:49:06,202 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:49:06,249 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc715bcd207e915d76dc8 HTTP/1.1" 200 None 2026-03-08 00:49:06,251 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:49:06,309 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc037bcd207e915d76bc8 HTTP/1.1" 200 None 2026-03-08 00:49:06,311 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:49:06,371 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc715bcd207e915d76dc8 HTTP/1.1" 200 None 2026-03-08 00:49:06,373 - lava_callback - INFO - Completed processing callback for node 69acc715bcd207e915d76dc8 INFO: 172.21.0.1:53466 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:49:13,966 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:49:13,993 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:47608 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47620 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55526 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55536 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48580 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48102 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58484 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35250 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58706 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58720 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40418 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40420 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46170 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53708 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40852 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42088 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:51:14,384 - lava_callback - INFO - Received callback for node 69acc7fcbcd207e915d76e69 INFO: 172.21.0.1:49532 - "POST /node/69acc7fcbcd207e915d76e69 HTTP/1.0" 202 Accepted 2026-03-08 00:51:14,391 - lava_callback - INFO - Starting async job submit for node 69acc7fcbcd207e915d76e69 2026-03-08 00:51:14,396 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:51:14,423 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc7fcbcd207e915d76e69 HTTP/1.1" 200 None 2026-03-08 00:51:14,425 - lava_callback - DEBUG - Processing node 69acc7fcbcd207e915d76e69: baseline-arm 2026-03-08 00:51:14,433 - lava_callback - WARNING - Failed to upload log 2026-03-08 00:51:14,439 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:51:14,868 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:51:14,870 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69acc7fcbcd207e915d76e69/lava-logs.yaml 2026-03-08 00:51:14,871 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:51:14,997 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:51:14,998 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69acc7fcbcd207e915d76e69/lava_callback.json.gz 2026-03-08 00:51:15,003 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:51:15,023 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc7fcbcd207e915d76e69 HTTP/1.1" 200 None 2026-03-08 00:51:15,025 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:51:15,038 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:51:15,040 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:51:15,060 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc7fcbcd207e915d76e69 HTTP/1.1" 200 None 2026-03-08 00:51:15,065 - lava_callback - INFO - Completed processing callback for node 69acc7fcbcd207e915d76e69 INFO: 172.21.0.1:49542 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:49552 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33038 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33044 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44458 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:51:43,996 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:51:44,029 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:60400 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:51:53,660 - lava_callback - INFO - Received callback for node 69acc7fdbcd207e915d76e6c INFO: 172.21.0.1:36310 - "POST /node/69acc7fdbcd207e915d76e6c HTTP/1.0" 202 Accepted 2026-03-08 00:51:53,711 - lava_callback - INFO - Starting async job submit for node 69acc7fdbcd207e915d76e6c 2026-03-08 00:51:53,731 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:51:53,755 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc7fdbcd207e915d76e6c HTTP/1.1" 200 None 2026-03-08 00:51:53,756 - lava_callback - DEBUG - Processing node 69acc7fdbcd207e915d76e6c: baseline-arm 2026-03-08 00:51:53,958 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:51:54,082 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:51:54,083 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69acc7fdbcd207e915d76e6c/log.txt.gz 2026-03-08 00:51:54,167 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:51:54,329 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:51:54,331 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69acc7fdbcd207e915d76e6c/lava-logs.yaml 2026-03-08 00:51:54,335 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:51:54,467 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:51:54,469 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69acc7fdbcd207e915d76e6c/lava_callback.json.gz 2026-03-08 00:51:54,494 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:51:54,535 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc7fdbcd207e915d76e6c HTTP/1.1" 200 None 2026-03-08 00:51:54,537 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:51:54,565 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:51:54,567 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:51:54,593 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc7fdbcd207e915d76e6c HTTP/1.1" 200 None 2026-03-08 00:51:54,594 - lava_callback - INFO - Completed processing callback for node 69acc7fdbcd207e915d76e6c INFO: 172.21.0.1:36324 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36306 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:52:12,303 - lava_callback - INFO - Received callback for node 69acc800bcd207e915d76e87 INFO: 172.21.0.1:59016 - "POST /node/69acc800bcd207e915d76e87 HTTP/1.0" 202 Accepted 2026-03-08 00:52:12,349 - lava_callback - INFO - Starting async job submit for node 69acc800bcd207e915d76e87 2026-03-08 00:52:12,366 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:12,381 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc800bcd207e915d76e87 HTTP/1.1" 200 None 2026-03-08 00:52:12,382 - lava_callback - DEBUG - Processing node 69acc800bcd207e915d76e87: baseline-arm 2026-03-08 00:52:12,505 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:12,679 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:12,682 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69acc800bcd207e915d76e87/log.txt.gz 2026-03-08 00:52:12,759 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:12,904 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:12,906 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69acc800bcd207e915d76e87/lava-logs.yaml 2026-03-08 00:52:12,910 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:13,017 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:13,018 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69acc800bcd207e915d76e87/lava_callback.json.gz 2026-03-08 00:52:13,036 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:13,050 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc800bcd207e915d76e87 HTTP/1.1" 200 None 2026-03-08 00:52:13,052 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:13,065 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:52:13,066 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:13,106 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc800bcd207e915d76e87 HTTP/1.1" 200 None 2026-03-08 00:52:13,112 - lava_callback - INFO - Completed processing callback for node 69acc800bcd207e915d76e87 2026-03-08 00:52:14,031 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:14,053 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:59026 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59032 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58074 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:52:26,006 - lava_callback - INFO - Received callback for node 69acc7fbbcd207e915d76e66 INFO: 172.21.0.1:58076 - "POST /node/69acc7fbbcd207e915d76e66 HTTP/1.0" 202 Accepted 2026-03-08 00:52:26,012 - lava_callback - INFO - Starting async job submit for node 69acc7fbbcd207e915d76e66 2026-03-08 00:52:26,033 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:26,056 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc7fbbcd207e915d76e66 HTTP/1.1" 200 None 2026-03-08 00:52:26,057 - lava_callback - DEBUG - Processing node 69acc7fbbcd207e915d76e66: baseline-arm 2026-03-08 00:52:26,323 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:26,457 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:26,458 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69acc7fbbcd207e915d76e66/log.txt.gz INFO: 172.21.0.1:58086 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:52:26,579 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:26,761 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:26,763 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69acc7fbbcd207e915d76e66/lava-logs.yaml 2026-03-08 00:52:26,768 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:26,948 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:26,950 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69acc7fbbcd207e915d76e66/lava_callback.json.gz 2026-03-08 00:52:26,972 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:26,986 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc7fbbcd207e915d76e66 HTTP/1.1" 200 None 2026-03-08 00:52:26,987 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:27,001 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:52:27,003 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:27,027 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc7fbbcd207e915d76e66 HTTP/1.1" 200 None 2026-03-08 00:52:27,032 - lava_callback - INFO - Completed processing callback for node 69acc7fbbcd207e915d76e66 2026-03-08 00:52:34,767 - lava_callback - INFO - Received callback for node 69acc7ffbcd207e915d76e84 INFO: 172.21.0.1:44176 - "POST /node/69acc7ffbcd207e915d76e84 HTTP/1.0" 202 Accepted 2026-03-08 00:52:34,777 - lava_callback - INFO - Starting async job submit for node 69acc7ffbcd207e915d76e84 2026-03-08 00:52:34,817 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:34,839 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc7ffbcd207e915d76e84 HTTP/1.1" 200 None 2026-03-08 00:52:34,841 - lava_callback - DEBUG - Processing node 69acc7ffbcd207e915d76e84: baseline-arm 2026-03-08 00:52:35,120 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:35,254 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:35,257 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69acc7ffbcd207e915d76e84/log.txt.gz 2026-03-08 00:52:35,389 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:35,587 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:35,589 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69acc7ffbcd207e915d76e84/lava-logs.yaml 2026-03-08 00:52:35,595 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:35,734 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:35,736 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69acc7ffbcd207e915d76e84/lava_callback.json.gz 2026-03-08 00:52:35,758 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:35,775 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc7ffbcd207e915d76e84 HTTP/1.1" 200 None 2026-03-08 00:52:35,777 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:35,797 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:52:35,800 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:35,830 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc7ffbcd207e915d76e84 HTTP/1.1" 200 None 2026-03-08 00:52:35,833 - lava_callback - INFO - Completed processing callback for node 69acc7ffbcd207e915d76e84 INFO: 172.21.0.1:44182 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:52:44,055 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:44,072 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:60252 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:52:52,113 - lava_callback - INFO - Received callback for node 69acc80abcd207e915d76e97 INFO: 172.21.0.1:35670 - "POST /node/69acc80abcd207e915d76e97 HTTP/1.0" 202 Accepted 2026-03-08 00:52:52,124 - lava_callback - INFO - Starting async job submit for node 69acc80abcd207e915d76e97 2026-03-08 00:52:52,169 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:52,193 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc80abcd207e915d76e97 HTTP/1.1" 200 None 2026-03-08 00:52:52,194 - lava_callback - DEBUG - Processing node 69acc80abcd207e915d76e97: baseline-arm-broonie 2026-03-08 00:52:52,260 - lava_callback - INFO - Received callback for node 69aa255f8c4d487a0e751ab8 2026-03-08 00:52:52,269 - lava_callback - INFO - Starting async job submit for node 69aa255f8c4d487a0e751ab8 INFO: 172.21.0.1:35680 - "POST /node/69aa255f8c4d487a0e751ab8 HTTP/1.0" 202 Accepted 2026-03-08 00:52:52,287 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:52,368 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa255f8c4d487a0e751ab8 HTTP/1.1" 200 None 2026-03-08 00:52:52,370 - lava_callback - DEBUG - Processing node 69aa255f8c4d487a0e751ab8: baseline-arm64-baylibre 2026-03-08 00:52:52,370 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa255f8c4d487a0e751ab8 2026-03-08 00:52:52,371 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:52,438 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa255f8c4d487a0e751ab8 HTTP/1.1" 200 None 2026-03-08 00:52:52,441 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:52,516 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa1d4a8c4d487a0e751161 HTTP/1.1" 200 None 2026-03-08 00:52:52,529 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:52,700 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa255f8c4d487a0e751ab8 HTTP/1.1" 200 None 2026-03-08 00:52:52,717 - lava_callback - ERROR - Error processing callback for node 69aa255f8c4d487a0e751ab8: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-08 00:52:52,731 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:53,406 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:53,409 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69acc80abcd207e915d76e97/log.txt.gz 2026-03-08 00:52:53,498 - lava_callback - INFO - Received callback for node 69acc80bbcd207e915d76e9a INFO: 172.21.0.1:35688 - "POST /node/69acc80bbcd207e915d76e9a HTTP/1.0" 202 Accepted 2026-03-08 00:52:53,522 - lava_callback - INFO - Starting async job submit for node 69acc80bbcd207e915d76e9a 2026-03-08 00:52:53,615 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:53,693 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc80bbcd207e915d76e9a HTTP/1.1" 200 None 2026-03-08 00:52:53,695 - lava_callback - DEBUG - Processing node 69acc80bbcd207e915d76e9a: baseline-arm-broonie 2026-03-08 00:52:53,718 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:54,138 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:54,553 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:54,556 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69acc80abcd207e915d76e97/lava-logs.yaml 2026-03-08 00:52:54,564 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:54,585 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:54,587 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69acc80bbcd207e915d76e9a/log.txt.gz 2026-03-08 00:52:54,818 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:54,923 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:54,926 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69acc80abcd207e915d76e97/lava_callback.json.gz 2026-03-08 00:52:54,965 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:55,000 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc80abcd207e915d76e97 HTTP/1.1" 200 None 2026-03-08 00:52:55,003 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:55,039 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:52:55,042 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:55,082 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc80abcd207e915d76e97 HTTP/1.1" 200 None 2026-03-08 00:52:55,083 - lava_callback - INFO - Completed processing callback for node 69acc80abcd207e915d76e97 2026-03-08 00:52:55,724 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:55,727 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69acc80bbcd207e915d76e9a/lava-logs.yaml 2026-03-08 00:52:55,741 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:55,994 - lava_callback - INFO - Received callback for node 69acc81dbcd207e915d76ef3 INFO: 172.21.0.1:35694 - "POST /node/69acc81dbcd207e915d76ef3 HTTP/1.0" 202 Accepted 2026-03-08 00:52:56,004 - lava_callback - INFO - Starting async job submit for node 69acc81dbcd207e915d76ef3 2026-03-08 00:52:56,049 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:56,094 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc81dbcd207e915d76ef3 HTTP/1.1" 200 None 2026-03-08 00:52:56,096 - lava_callback - DEBUG - Processing node 69acc81dbcd207e915d76ef3: baseline-arm-pengutronix 2026-03-08 00:52:56,166 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:56,168 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69acc80bbcd207e915d76e9a/lava_callback.json.gz 2026-03-08 00:52:56,239 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:56,305 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc80bbcd207e915d76e9a HTTP/1.1" 200 None 2026-03-08 00:52:56,307 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:56,385 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:52:56,403 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:56,436 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:56,463 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc80bbcd207e915d76e9a HTTP/1.1" 200 None 2026-03-08 00:52:56,465 - lava_callback - INFO - Completed processing callback for node 69acc80bbcd207e915d76e9a INFO: 172.21.0.1:35702 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:52:57,009 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:57,012 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69acc81dbcd207e915d76ef3/log.txt.gz 2026-03-08 00:52:57,179 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:57,250 - lava_callback - INFO - Received callback for node 69acc81ebcd207e915d76ef6 INFO: 172.21.0.1:35708 - "POST /node/69acc81ebcd207e915d76ef6 HTTP/1.0" 202 Accepted 2026-03-08 00:52:57,259 - lava_callback - INFO - Starting async job submit for node 69acc81ebcd207e915d76ef6 2026-03-08 00:52:57,304 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:57,332 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc81ebcd207e915d76ef6 HTTP/1.1" 200 None 2026-03-08 00:52:57,334 - lava_callback - DEBUG - Processing node 69acc81ebcd207e915d76ef6: baseline-arm-pengutronix 2026-03-08 00:52:57,608 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:58,199 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:58,202 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69acc81ebcd207e915d76ef6/log.txt.gz 2026-03-08 00:52:58,289 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:58,303 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69acc81dbcd207e915d76ef3/lava-logs.yaml 2026-03-08 00:52:58,325 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:58,394 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:52:59,049 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:59,051 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69acc81dbcd207e915d76ef3/lava_callback.json.gz 2026-03-08 00:52:59,094 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:59,120 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc81dbcd207e915d76ef3 HTTP/1.1" 200 None 2026-03-08 00:52:59,122 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:59,144 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:52:59,147 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:52:59,190 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc81dbcd207e915d76ef3 HTTP/1.1" 200 None 2026-03-08 00:52:59,192 - lava_callback - INFO - Completed processing callback for node 69acc81dbcd207e915d76ef3 2026-03-08 00:52:59,307 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:52:59,310 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69acc81ebcd207e915d76ef6/lava-logs.yaml 2026-03-08 00:52:59,316 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:53:00,087 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:53:00,089 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69acc81ebcd207e915d76ef6/lava_callback.json.gz 2026-03-08 00:53:00,131 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:00,155 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc81ebcd207e915d76ef6 HTTP/1.1" 200 None 2026-03-08 00:53:00,157 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:00,181 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:53:00,184 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:00,223 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc81ebcd207e915d76ef6 HTTP/1.1" 200 None 2026-03-08 00:53:00,225 - lava_callback - INFO - Completed processing callback for node 69acc81ebcd207e915d76ef6 2026-03-08 00:53:02,525 - lava_callback - INFO - Received callback for node 69acc80dbcd207e915d76e9d INFO: 172.21.0.1:43320 - "POST /node/69acc80dbcd207e915d76e9d HTTP/1.0" 202 Accepted 2026-03-08 00:53:02,544 - lava_callback - INFO - Starting async job submit for node 69acc80dbcd207e915d76e9d 2026-03-08 00:53:02,596 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:02,618 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc80dbcd207e915d76e9d HTTP/1.1" 200 None 2026-03-08 00:53:02,619 - lava_callback - DEBUG - Processing node 69acc80dbcd207e915d76e9d: baseline-arm-broonie 2026-03-08 00:53:03,067 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:53:03,484 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:53:03,486 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69acc80dbcd207e915d76e9d/log.txt.gz 2026-03-08 00:53:03,791 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:53:04,378 - lava_callback - INFO - Received callback for node 69acc815bcd207e915d76ebc INFO: 172.21.0.1:43324 - "POST /node/69acc815bcd207e915d76ebc HTTP/1.0" 202 Accepted 2026-03-08 00:53:04,389 - lava_callback - INFO - Starting async job submit for node 69acc815bcd207e915d76ebc 2026-03-08 00:53:04,416 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:53:04,419 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69acc80dbcd207e915d76e9d/lava-logs.yaml 2026-03-08 00:53:04,442 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:53:04,464 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:04,509 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc815bcd207e915d76ebc HTTP/1.1" 200 None 2026-03-08 00:53:04,510 - lava_callback - DEBUG - Processing node 69acc815bcd207e915d76ebc: baseline-arm-clabbe 2026-03-08 00:53:04,825 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:53:05,186 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:53:05,189 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69acc80dbcd207e915d76e9d/lava_callback.json.gz 2026-03-08 00:53:05,238 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:05,253 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:53:05,255 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acc815bcd207e915d76ebc/log.txt.gz 2026-03-08 00:53:05,289 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc80dbcd207e915d76e9d HTTP/1.1" 200 None 2026-03-08 00:53:05,291 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:05,340 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:53:05,358 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:05,483 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc80dbcd207e915d76e9d HTTP/1.1" 200 None 2026-03-08 00:53:05,485 - lava_callback - INFO - Completed processing callback for node 69acc80dbcd207e915d76e9d 2026-03-08 00:53:05,598 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:53:05,807 - lava_callback - INFO - Received callback for node 69acc821bcd207e915d76f00 INFO: 172.21.0.1:43326 - "POST /node/69acc821bcd207e915d76f00 HTTP/1.0" 202 Accepted 2026-03-08 00:53:05,856 - lava_callback - INFO - Starting async job submit for node 69acc821bcd207e915d76f00 2026-03-08 00:53:05,903 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:05,930 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc821bcd207e915d76f00 HTTP/1.1" 200 None 2026-03-08 00:53:05,931 - lava_callback - DEBUG - Processing node 69acc821bcd207e915d76f00: baseline-arm-pengutronix 2026-03-08 00:53:06,174 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:43340 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:53:06,711 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:53:06,713 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69acc821bcd207e915d76f00/log.txt.gz 2026-03-08 00:53:06,786 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:53:06,790 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acc815bcd207e915d76ebc/lava-logs.yaml 2026-03-08 00:53:06,803 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:53:06,855 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:53:07,606 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:53:07,608 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acc815bcd207e915d76ebc/lava_callback.json.gz 2026-03-08 00:53:07,653 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:07,680 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:53:07,683 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc815bcd207e915d76ebc HTTP/1.1" 200 None 2026-03-08 00:53:07,685 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:07,686 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69acc821bcd207e915d76f00/lava-logs.yaml 2026-03-08 00:53:07,690 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:53:07,709 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:53:07,711 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:07,767 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc815bcd207e915d76ebc HTTP/1.1" 200 None 2026-03-08 00:53:07,769 - lava_callback - INFO - Completed processing callback for node 69acc815bcd207e915d76ebc 2026-03-08 00:53:08,364 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:53:08,366 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69acc821bcd207e915d76f00/lava_callback.json.gz 2026-03-08 00:53:08,423 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:08,451 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc821bcd207e915d76f00 HTTP/1.1" 200 None 2026-03-08 00:53:08,454 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:08,495 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:53:08,498 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:08,542 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc821bcd207e915d76f00 HTTP/1.1" 200 None 2026-03-08 00:53:08,544 - lava_callback - INFO - Completed processing callback for node 69acc821bcd207e915d76f00 2026-03-08 00:53:10,052 - lava_callback - INFO - Received callback for node 69acc811bcd207e915d76eae INFO: 172.21.0.1:41510 - "POST /node/69acc811bcd207e915d76eae HTTP/1.0" 202 Accepted 2026-03-08 00:53:10,062 - lava_callback - INFO - Starting async job submit for node 69acc811bcd207e915d76eae 2026-03-08 00:53:10,104 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:10,136 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc811bcd207e915d76eae HTTP/1.1" 200 None 2026-03-08 00:53:10,138 - lava_callback - DEBUG - Processing node 69acc811bcd207e915d76eae: baseline-arm-clabbe 2026-03-08 00:53:10,650 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:53:11,441 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:53:11,444 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acc811bcd207e915d76eae/log.txt.gz 2026-03-08 00:53:11,745 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:53:12,380 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:53:12,383 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acc811bcd207e915d76eae/lava-logs.yaml 2026-03-08 00:53:12,391 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:53:13,054 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:53:13,056 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acc811bcd207e915d76eae/lava_callback.json.gz 2026-03-08 00:53:13,098 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:13,124 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc811bcd207e915d76eae HTTP/1.1" 200 None 2026-03-08 00:53:13,126 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:13,156 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:53:13,159 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:13,206 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc811bcd207e915d76eae HTTP/1.1" 200 None 2026-03-08 00:53:13,208 - lava_callback - INFO - Completed processing callback for node 69acc811bcd207e915d76eae 2026-03-08 00:53:14,075 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:14,117 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 2026-03-08 00:53:14,412 - lava_callback - INFO - Received callback for node 69aa25718c4d487a0e751b04 INFO: 172.21.0.1:41526 - "POST /node/69aa25718c4d487a0e751b04 HTTP/1.0" 202 Accepted 2026-03-08 00:53:14,421 - lava_callback - INFO - Starting async job submit for node 69aa25718c4d487a0e751b04 2026-03-08 00:53:14,423 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:14,453 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa25718c4d487a0e751b04 HTTP/1.1" 200 None 2026-03-08 00:53:14,455 - lava_callback - DEBUG - Processing node 69aa25718c4d487a0e751b04: baseline-arm64-clabbe 2026-03-08 00:53:14,455 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa25718c4d487a0e751b04 2026-03-08 00:53:14,456 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:14,481 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa25718c4d487a0e751b04 HTTP/1.1" 200 None 2026-03-08 00:53:14,483 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:14,507 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa1d4a8c4d487a0e751161 HTTP/1.1" 200 None 2026-03-08 00:53:14,509 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:14,544 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa25718c4d487a0e751b04 HTTP/1.1" 200 None 2026-03-08 00:53:14,545 - lava_callback - ERROR - Error processing callback for node 69aa25718c4d487a0e751b04: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:41536 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:41538 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:53:17,177 - lava_callback - INFO - Received callback for node 69acc81abcd207e915d76ec9 INFO: 172.21.0.1:41554 - "POST /node/69acc81abcd207e915d76ec9 HTTP/1.0" 202 Accepted 2026-03-08 00:53:17,186 - lava_callback - INFO - Starting async job submit for node 69acc81abcd207e915d76ec9 2026-03-08 00:53:17,237 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:17,266 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc81abcd207e915d76ec9 HTTP/1.1" 200 None 2026-03-08 00:53:17,268 - lava_callback - DEBUG - Processing node 69acc81abcd207e915d76ec9: baseline-arm-kontron 2026-03-08 00:53:17,600 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:53:18,309 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:53:18,312 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-kontron-69acc81abcd207e915d76ec9/log.txt.gz 2026-03-08 00:53:18,505 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:53:19,427 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:53:19,429 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-kontron-69acc81abcd207e915d76ec9/lava-logs.yaml 2026-03-08 00:53:19,436 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:53:19,866 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:53:19,868 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-kontron-69acc81abcd207e915d76ec9/lava_callback.json.gz 2026-03-08 00:53:19,908 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:19,938 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc81abcd207e915d76ec9 HTTP/1.1" 200 None 2026-03-08 00:53:19,940 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:19,969 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:53:19,971 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:20,021 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc81abcd207e915d76ec9 HTTP/1.1" 200 None 2026-03-08 00:53:20,023 - lava_callback - INFO - Completed processing callback for node 69acc81abcd207e915d76ec9 INFO: 172.21.0.1:55936 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55948 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:53:34,192 - lava_callback - INFO - Received callback for node 69aa25798c4d487a0e751b20 INFO: 172.21.0.1:58514 - "POST /node/69aa25798c4d487a0e751b20 HTTP/1.0" 202 Accepted 2026-03-08 00:53:34,201 - lava_callback - INFO - Starting async job submit for node 69aa25798c4d487a0e751b20 2026-03-08 00:53:34,206 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:34,230 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa25798c4d487a0e751b20 HTTP/1.1" 200 None 2026-03-08 00:53:34,231 - lava_callback - DEBUG - Processing node 69aa25798c4d487a0e751b20: baseline-arm64-clabbe 2026-03-08 00:53:34,231 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa25798c4d487a0e751b20 2026-03-08 00:53:34,232 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:34,258 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa25798c4d487a0e751b20 HTTP/1.1" 200 None 2026-03-08 00:53:34,261 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:34,285 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa1d4a8c4d487a0e751161 HTTP/1.1" 200 None 2026-03-08 00:53:34,287 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:34,326 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa25798c4d487a0e751b20 HTTP/1.1" 200 None 2026-03-08 00:53:34,328 - lava_callback - ERROR - Error processing callback for node 69aa25798c4d487a0e751b20: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:58522 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:53:44,120 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:53:44,148 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:34904 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39012 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37990 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33664 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:33672 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59242 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59248 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:54:32,667 - lava_callback - INFO - Received callback for node 69acc839bcd207e915d76f1e INFO: 172.21.0.1:54542 - "POST /node/69acc839bcd207e915d76f1e HTTP/1.0" 202 Accepted 2026-03-08 00:54:32,678 - lava_callback - INFO - Starting async job submit for node 69acc839bcd207e915d76f1e 2026-03-08 00:54:32,721 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:54:32,754 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc839bcd207e915d76f1e HTTP/1.1" 200 None 2026-03-08 00:54:32,755 - lava_callback - DEBUG - Processing node 69acc839bcd207e915d76f1e: ltp-dio 2026-03-08 00:54:33,481 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:54:34,466 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:54:34,469 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-dio-69acc839bcd207e915d76f1e/log.txt.gz 2026-03-08 00:54:34,856 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:54:36,505 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:54:36,508 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-dio-69acc839bcd207e915d76f1e/lava-logs.yaml INFO: 172.21.0.1:54544 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:54:36,521 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:54:37,657 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:54:37,659 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-dio-69acc839bcd207e915d76f1e/lava_callback.json.gz 2026-03-08 00:54:37,701 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:54:37,733 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc839bcd207e915d76f1e HTTP/1.1" 200 None 2026-03-08 00:54:37,736 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:54:37,763 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:54:37,766 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:54:37,804 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc839bcd207e915d76f1e HTTP/1.1" 200 None 2026-03-08 00:54:37,806 - lava_callback - INFO - Completed processing callback for node 69acc839bcd207e915d76f1e 2026-03-08 00:54:44,151 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:54:44,177 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:37716 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:54:53,192 - lava_callback - INFO - Received callback for node 69acc812bcd207e915d76eb2 INFO: 172.21.0.1:40752 - "POST /node/69acc812bcd207e915d76eb2 HTTP/1.0" 202 Accepted 2026-03-08 00:54:53,208 - lava_callback - INFO - Starting async job submit for node 69acc812bcd207e915d76eb2 2026-03-08 00:54:53,299 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:54:53,326 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc812bcd207e915d76eb2 HTTP/1.1" 200 None 2026-03-08 00:54:53,328 - lava_callback - DEBUG - Processing node 69acc812bcd207e915d76eb2: baseline-arm-clabbe 2026-03-08 00:54:53,692 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:54:54,658 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:54:54,660 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acc812bcd207e915d76eb2/log.txt.gz 2026-03-08 00:54:54,893 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:54:56,225 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:54:56,227 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acc812bcd207e915d76eb2/lava-logs.yaml 2026-03-08 00:54:56,234 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:40760 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:54:57,361 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:54:57,363 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acc812bcd207e915d76eb2/lava_callback.json.gz 2026-03-08 00:54:57,407 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:54:57,430 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc812bcd207e915d76eb2 HTTP/1.1" 200 None 2026-03-08 00:54:57,432 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:54:57,470 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:54:57,472 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:54:57,513 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc812bcd207e915d76eb2 HTTP/1.1" 200 None 2026-03-08 00:54:57,515 - lava_callback - INFO - Completed processing callback for node 69acc812bcd207e915d76eb2 INFO: 172.21.0.1:54534 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:55:14,180 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:55:14,246 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:37270 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:37276 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43732 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43746 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:55:35,068 - lava_callback - INFO - Received callback for node 69acc8fabcd207e915d77007 INFO: 172.21.0.1:42420 - "POST /node/69acc8fabcd207e915d77007 HTTP/1.0" 202 Accepted 2026-03-08 00:55:35,077 - lava_callback - INFO - Starting async job submit for node 69acc8fabcd207e915d77007 2026-03-08 00:55:35,087 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:55:35,111 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc8fabcd207e915d77007 HTTP/1.1" 200 None 2026-03-08 00:55:35,112 - lava_callback - DEBUG - Processing node 69acc8fabcd207e915d77007: baseline-arm 2026-03-08 00:55:35,121 - lava_callback - WARNING - Failed to upload log 2026-03-08 00:55:35,127 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:55:36,058 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:55:36,061 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69acc8fabcd207e915d77007/lava-logs.yaml 2026-03-08 00:55:36,062 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:42424 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:55:36,900 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:55:36,902 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69acc8fabcd207e915d77007/lava_callback.json.gz 2026-03-08 00:55:36,906 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:55:36,931 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc8fabcd207e915d77007 HTTP/1.1" 200 None 2026-03-08 00:55:36,933 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:55:36,959 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:55:36,961 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:55:36,993 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc8fabcd207e915d77007 HTTP/1.1" 200 None 2026-03-08 00:55:36,998 - lava_callback - INFO - Completed processing callback for node 69acc8fabcd207e915d77007 2026-03-08 00:55:44,249 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:55:44,275 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:55808 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38528 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36100 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32856 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:32868 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51152 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:56:23,355 - lava_callback - INFO - Received callback for node 69acc847bcd207e915d76f4c INFO: 172.21.0.1:51156 - "POST /node/69acc847bcd207e915d76f4c HTTP/1.0" 202 Accepted 2026-03-08 00:56:23,370 - lava_callback - INFO - Starting async job submit for node 69acc847bcd207e915d76f4c 2026-03-08 00:56:23,423 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:56:23,446 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc847bcd207e915d76f4c HTTP/1.1" 200 None 2026-03-08 00:56:23,447 - lava_callback - DEBUG - Processing node 69acc847bcd207e915d76f4c: ltp-syscalls 2026-03-08 00:56:24,728 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:56:25,193 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:56:25,196 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-syscalls-69acc847bcd207e915d76f4c/log.txt.gz 2026-03-08 00:56:25,886 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:51166 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:56:26,740 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:56:26,743 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-syscalls-69acc847bcd207e915d76f4c/lava-logs.yaml 2026-03-08 00:56:26,768 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:56:27,191 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:56:27,193 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-syscalls-69acc847bcd207e915d76f4c/lava_callback.json.gz 2026-03-08 00:56:27,247 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:56:27,286 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc847bcd207e915d76f4c HTTP/1.1" 200 None 2026-03-08 00:56:27,288 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:56:27,313 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:56:27,315 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:56:27,361 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc847bcd207e915d76f4c HTTP/1.1" 200 None 2026-03-08 00:56:27,363 - lava_callback - INFO - Completed processing callback for node 69acc847bcd207e915d76f4c 2026-03-08 00:56:36,283 - lava_callback - INFO - Received callback for node 69acc816bcd207e915d76ebf INFO: 172.21.0.1:50752 - "POST /node/69acc816bcd207e915d76ebf HTTP/1.0" 202 Accepted 2026-03-08 00:56:36,292 - lava_callback - INFO - Starting async job submit for node 69acc816bcd207e915d76ebf 2026-03-08 00:56:36,331 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:56:36,362 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc816bcd207e915d76ebf HTTP/1.1" 200 None 2026-03-08 00:56:36,363 - lava_callback - DEBUG - Processing node 69acc816bcd207e915d76ebf: baseline-arm-clabbe INFO: 172.21.0.1:50760 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:56:36,756 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:56:37,060 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:56:37,063 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acc816bcd207e915d76ebf/log.txt.gz 2026-03-08 00:56:37,285 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:56:37,821 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:56:37,823 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acc816bcd207e915d76ebf/lava-logs.yaml 2026-03-08 00:56:37,830 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:56:38,257 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:56:38,259 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acc816bcd207e915d76ebf/lava_callback.json.gz 2026-03-08 00:56:38,316 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:56:38,346 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc816bcd207e915d76ebf HTTP/1.1" 200 None 2026-03-08 00:56:38,348 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:56:38,372 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:56:38,375 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:56:38,414 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc816bcd207e915d76ebf HTTP/1.1" 200 None 2026-03-08 00:56:38,417 - lava_callback - INFO - Completed processing callback for node 69acc816bcd207e915d76ebf 2026-03-08 00:56:44,277 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:56:44,293 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:50686 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57672 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45050 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43376 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:43378 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44262 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44268 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44606 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59818 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:57:52,934 - lava_callback - INFO - Received callback for node 69acc501bcd207e915d76ce2 INFO: 172.21.0.1:41350 - "POST /node/69acc501bcd207e915d76ce2 HTTP/1.0" 202 Accepted 2026-03-08 00:57:52,942 - lava_callback - INFO - Starting async job submit for node 69acc501bcd207e915d76ce2 2026-03-08 00:57:53,035 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:57:53,052 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc501bcd207e915d76ce2 HTTP/1.1" 200 None 2026-03-08 00:57:53,053 - lava_callback - DEBUG - Processing node 69acc501bcd207e915d76ce2: ltp-smoke 2026-03-08 00:57:53,711 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:57:53,922 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:57:53,924 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69acc501bcd207e915d76ce2/log.txt.gz 2026-03-08 00:57:54,273 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:57:54,496 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:57:54,498 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69acc501bcd207e915d76ce2/lava-logs.yaml 2026-03-08 00:57:54,512 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:57:54,760 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:57:54,762 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69acc501bcd207e915d76ce2/lava_callback.json.gz 2026-03-08 00:57:54,912 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:57:54,932 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc501bcd207e915d76ce2 HTTP/1.1" 200 None 2026-03-08 00:57:54,935 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:57:54,949 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc012bcd207e915d76b9c HTTP/1.1" 200 None 2026-03-08 00:57:54,951 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:57:54,988 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc501bcd207e915d76ce2 HTTP/1.1" 200 None 2026-03-08 00:57:55,148 - lava_callback - INFO - Completed processing callback for node 69acc501bcd207e915d76ce2 INFO: 172.21.0.1:41356 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35348 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:58:14,295 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:58:14,314 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:38792 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:38794 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51226 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51242 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:58:31,411 - lava_callback - INFO - Received callback for node 69acc817bcd207e915d76ec2 INFO: 172.21.0.1:33342 - "POST /node/69acc817bcd207e915d76ec2 HTTP/1.0" 202 Accepted 2026-03-08 00:58:31,418 - lava_callback - INFO - Starting async job submit for node 69acc817bcd207e915d76ec2 2026-03-08 00:58:31,440 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:58:31,454 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc817bcd207e915d76ec2 HTTP/1.1" 200 None 2026-03-08 00:58:31,455 - lava_callback - DEBUG - Processing node 69acc817bcd207e915d76ec2: baseline-arm-clabbe 2026-03-08 00:58:31,662 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:58:31,813 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:58:31,815 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acc817bcd207e915d76ec2/log.txt.gz 2026-03-08 00:58:31,950 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:58:32,154 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:58:32,156 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acc817bcd207e915d76ec2/lava-logs.yaml 2026-03-08 00:58:32,162 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 00:58:32,300 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 00:58:32,302 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acc817bcd207e915d76ec2/lava_callback.json.gz 2026-03-08 00:58:32,324 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:58:32,338 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc817bcd207e915d76ec2 HTTP/1.1" 200 None 2026-03-08 00:58:32,340 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:58:32,353 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 00:58:32,355 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:58:32,379 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc817bcd207e915d76ec2 HTTP/1.1" 200 None 2026-03-08 00:58:32,380 - lava_callback - INFO - Completed processing callback for node 69acc817bcd207e915d76ec2 INFO: 172.21.0.1:33356 - "GET / HTTP/1.0" 200 OK 2026-03-08 00:58:44,316 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 00:58:44,332 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:34900 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44952 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38268 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46080 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46082 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45210 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45216 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42036 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39848 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44368 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53060 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54748 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54750 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42578 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42594 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58342 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55048 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44602 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56976 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46156 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46172 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34374 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34388 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51022 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37224 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:01:52,665 - lava_callback - INFO - Received callback for node 69aa27728c4d487a0e751d63 INFO: 172.21.0.1:57030 - "POST /node/69aa27728c4d487a0e751d63 HTTP/1.0" 202 Accepted 2026-03-08 01:01:52,671 - lava_callback - INFO - Starting async job submit for node 69aa27728c4d487a0e751d63 2026-03-08 01:01:52,672 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:01:52,687 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa27728c4d487a0e751d63 HTTP/1.1" 200 None 2026-03-08 01:01:52,688 - lava_callback - DEBUG - Processing node 69aa27728c4d487a0e751d63: baseline-arm64-baylibre 2026-03-08 01:01:52,688 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa27728c4d487a0e751d63 2026-03-08 01:01:52,689 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:01:52,703 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa27728c4d487a0e751d63 HTTP/1.1" 200 None 2026-03-08 01:01:52,704 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:01:52,718 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a70fafe31b554803095f11 HTTP/1.1" 200 None 2026-03-08 01:01:52,720 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:01:52,742 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa27728c4d487a0e751d63 HTTP/1.1" 200 None 2026-03-08 01:01:52,744 - lava_callback - ERROR - Error processing callback for node 69aa27728c4d487a0e751d63: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:57036 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40084 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35546 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:35558 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48470 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48482 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60016 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57456 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60000 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59136 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:03:13,511 - lava_callback - INFO - Received callback for node 69acc715bcd207e915d76dcc INFO: 172.21.0.1:42812 - "POST /node/69acc715bcd207e915d76dcc HTTP/1.0" 202 Accepted 2026-03-08 01:03:13,518 - lava_callback - INFO - Starting async job submit for node 69acc715bcd207e915d76dcc 2026-03-08 01:03:13,562 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:03:13,577 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc715bcd207e915d76dcc HTTP/1.1" 200 None 2026-03-08 01:03:13,577 - lava_callback - DEBUG - Processing node 69acc715bcd207e915d76dcc: baseline-x86 2026-03-08 01:03:14,304 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:03:14,512 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:03:14,514 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69acc715bcd207e915d76dcc/log.txt.gz 2026-03-08 01:03:14,951 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:42824 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 01:03:15,191 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:03:15,194 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69acc715bcd207e915d76dcc/lava-logs.yaml 2026-03-08 01:03:15,212 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:03:15,394 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:03:15,396 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69acc715bcd207e915d76dcc/lava_callback.json.gz DEBUG: Setup failed for 69acc715bcd207e915d76dcc. Transitting job node result: incomplete -> fail 2026-03-08 01:03:15,487 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:03:15,512 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc715bcd207e915d76dcc HTTP/1.1" 200 None 2026-03-08 01:03:15,514 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:03:15,527 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc037bcd207e915d76bc8 HTTP/1.1" 200 None 2026-03-08 01:03:15,529 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:03:15,557 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc715bcd207e915d76dcc HTTP/1.1" 200 None 2026-03-08 01:03:15,645 - lava_callback - INFO - Completed processing callback for node 69acc715bcd207e915d76dcc INFO: 172.21.0.1:42834 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38106 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38110 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41396 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:03:44,335 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:03:44,373 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:39196 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35664 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55394 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49342 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:49350 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38038 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38046 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34260 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37832 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50318 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53954 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38042 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:38052 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48280 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:05:21,613 - lava_callback - INFO - Received callback for node 69accb43bcd207e915d774f5 INFO: 172.21.0.1:48294 - "POST /node/69accb43bcd207e915d774f5 HTTP/1.0" 202 Accepted 2026-03-08 01:05:21,619 - lava_callback - INFO - Starting async job submit for node 69accb43bcd207e915d774f5 2026-03-08 01:05:21,622 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:05:21,636 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accb43bcd207e915d774f5 HTTP/1.1" 200 None 2026-03-08 01:05:21,637 - lava_callback - DEBUG - Processing node 69accb43bcd207e915d774f5: baseline-arm 2026-03-08 01:05:21,642 - lava_callback - WARNING - Failed to upload log 2026-03-08 01:05:21,645 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:05:21,792 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:05:21,794 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69accb43bcd207e915d774f5/lava-logs.yaml 2026-03-08 01:05:21,795 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:05:21,945 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:05:21,947 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69accb43bcd207e915d774f5/lava_callback.json.gz 2026-03-08 01:05:21,950 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:05:21,964 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accb43bcd207e915d774f5 HTTP/1.1" 200 None 2026-03-08 01:05:21,965 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:05:21,978 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 01:05:21,980 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:05:22,003 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accb43bcd207e915d774f5 HTTP/1.1" 200 None 2026-03-08 01:05:22,009 - lava_callback - INFO - Completed processing callback for node 69accb43bcd207e915d774f5 INFO: 172.21.0.1:48302 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36948 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:05:44,375 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:05:44,392 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:43830 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53392 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38782 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60144 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:60146 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57058 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57066 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47624 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33146 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44284 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42620 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48626 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48632 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33462 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33464 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37038 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:07:42,946 - lava_callback - INFO - Received callback for node 69acc7febcd207e915d76e6f INFO: 172.21.0.1:42992 - "POST /node/69acc7febcd207e915d76e6f HTTP/1.0" 202 Accepted 2026-03-08 01:07:42,953 - lava_callback - INFO - Starting async job submit for node 69acc7febcd207e915d76e6f 2026-03-08 01:07:43,006 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:07:43,022 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc7febcd207e915d76e6f HTTP/1.1" 200 None 2026-03-08 01:07:43,022 - lava_callback - DEBUG - Processing node 69acc7febcd207e915d76e6f: baseline-arm 2026-03-08 01:07:43,643 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:07:43,803 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:07:43,805 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69acc7febcd207e915d76e6f/log.txt.gz 2026-03-08 01:07:44,154 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:07:44,400 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:07:44,402 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69acc7febcd207e915d76e6f/lava-logs.yaml 2026-03-08 01:07:44,417 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:07:44,577 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:07:44,579 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69acc7febcd207e915d76e6f/lava_callback.json.gz DEBUG: Setup failed for 69acc7febcd207e915d76e6f. Transitting job node result: incomplete -> fail 2026-03-08 01:07:44,688 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:07:44,702 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc7febcd207e915d76e6f HTTP/1.1" 200 None 2026-03-08 01:07:44,704 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:07:44,742 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 01:07:44,744 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:07:44,770 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc7febcd207e915d76e6f HTTP/1.1" 200 None 2026-03-08 01:07:44,915 - lava_callback - INFO - Completed processing callback for node 69acc7febcd207e915d76e6f INFO: 172.21.0.1:43002 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47918 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41714 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:08:14,394 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:08:14,410 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:36162 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:36168 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35996 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35998 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45252 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45004 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42950 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41874 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46502 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46514 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59454 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59462 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49960 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56514 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43584 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60884 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:10:13,175 - lava_callback - INFO - Received callback for node 69aa29648c4d487a0e751e91 INFO: 172.21.0.1:48412 - "POST /node/69aa29648c4d487a0e751e91 HTTP/1.0" 202 Accepted 2026-03-08 01:10:13,180 - lava_callback - INFO - Starting async job submit for node 69aa29648c4d487a0e751e91 2026-03-08 01:10:13,181 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:10:13,196 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa29648c4d487a0e751e91 HTTP/1.1" 200 None 2026-03-08 01:10:13,197 - lava_callback - DEBUG - Processing node 69aa29648c4d487a0e751e91: baseline-arm64-baylibre 2026-03-08 01:10:13,197 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa29648c4d487a0e751e91 2026-03-08 01:10:13,198 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:10:13,213 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa29648c4d487a0e751e91 HTTP/1.1" 200 None 2026-03-08 01:10:13,214 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:10:13,227 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a70fafe31b554803095f11 HTTP/1.1" 200 None 2026-03-08 01:10:13,229 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:10:13,249 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa29648c4d487a0e751e91 HTTP/1.1" 200 None 2026-03-08 01:10:13,258 - lava_callback - ERROR - Error processing callback for node 69aa29648c4d487a0e751e91: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:48414 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48424 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47820 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47834 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53348 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:10:44,896 - lava_callback - INFO - Received callback for node 69acc80fbcd207e915d76ea8 INFO: 172.21.0.1:60970 - "POST /node/69acc80fbcd207e915d76ea8 HTTP/1.0" 202 Accepted 2026-03-08 01:10:44,902 - lava_callback - INFO - Starting async job submit for node 69acc80fbcd207e915d76ea8 2026-03-08 01:10:44,932 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:10:44,946 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc80fbcd207e915d76ea8 HTTP/1.1" 200 None 2026-03-08 01:10:44,947 - lava_callback - DEBUG - Processing node 69acc80fbcd207e915d76ea8: baseline-arm-clabbe 2026-03-08 01:10:45,145 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:10:45,303 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:10:45,305 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acc80fbcd207e915d76ea8/log.txt.gz 2026-03-08 01:10:45,423 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:10:45,656 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:10:45,658 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acc80fbcd207e915d76ea8/lava-logs.yaml 2026-03-08 01:10:45,662 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:10:45,774 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:10:45,776 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acc80fbcd207e915d76ea8/lava_callback.json.gz DEBUG: Setup failed for 69acc80fbcd207e915d76ea8. Transitting job node result: incomplete -> fail 2026-03-08 01:10:45,839 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:10:45,854 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc80fbcd207e915d76ea8 HTTP/1.1" 200 None 2026-03-08 01:10:45,856 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:10:45,872 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 01:10:45,874 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:10:45,903 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc80fbcd207e915d76ea8 HTTP/1.1" 200 None 2026-03-08 01:10:45,998 - lava_callback - INFO - Completed processing callback for node 69acc80fbcd207e915d76ea8 INFO: 172.21.0.1:60982 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56658 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35758 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:11:14,412 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:11:14,430 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:44618 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44626 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37690 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37698 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:11:33,243 - lava_callback - INFO - Received callback for node 69aa29b68c4d487a0e751eb4 INFO: 172.21.0.1:44338 - "POST /node/69aa29b68c4d487a0e751eb4 HTTP/1.0" 202 Accepted 2026-03-08 01:11:33,249 - lava_callback - INFO - Starting async job submit for node 69aa29b68c4d487a0e751eb4 2026-03-08 01:11:33,250 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:11:33,269 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa29b68c4d487a0e751eb4 HTTP/1.1" 200 None 2026-03-08 01:11:33,270 - lava_callback - DEBUG - Processing node 69aa29b68c4d487a0e751eb4: baseline-arm64-baylibre 2026-03-08 01:11:33,271 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa29b68c4d487a0e751eb4 2026-03-08 01:11:33,271 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:11:33,285 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa29b68c4d487a0e751eb4 HTTP/1.1" 200 None 2026-03-08 01:11:33,286 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:11:33,302 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a70fafe31b554803095f11 HTTP/1.1" 200 None 2026-03-08 01:11:33,304 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:11:33,324 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa29b68c4d487a0e751eb4 HTTP/1.1" 200 None 2026-03-08 01:11:33,333 - lava_callback - ERROR - Error processing callback for node 69aa29b68c4d487a0e751eb4: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:44346 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52512 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39086 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55534 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55146 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55154 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33826 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33834 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45446 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38758 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36394 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:13:01,383 - lava_callback - INFO - Received callback for node 69accd14bcd207e915d775de INFO: 172.21.0.1:60506 - "POST /node/69accd14bcd207e915d775de HTTP/1.0" 202 Accepted 2026-03-08 01:13:01,393 - lava_callback - INFO - Starting async job submit for node 69accd14bcd207e915d775de 2026-03-08 01:13:01,398 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:13:01,428 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accd14bcd207e915d775de HTTP/1.1" 200 None 2026-03-08 01:13:01,429 - lava_callback - DEBUG - Processing node 69accd14bcd207e915d775de: baseline-arm 2026-03-08 01:13:01,437 - lava_callback - WARNING - Failed to upload log 2026-03-08 01:13:01,442 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:13:01,625 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:13:01,627 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69accd14bcd207e915d775de/lava-logs.yaml 2026-03-08 01:13:01,628 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:13:01,747 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:13:01,749 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69accd14bcd207e915d775de/lava_callback.json.gz 2026-03-08 01:13:01,754 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:13:01,774 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accd14bcd207e915d775de HTTP/1.1" 200 None 2026-03-08 01:13:01,776 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:13:01,790 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 01:13:01,792 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:13:01,821 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accd14bcd207e915d775de HTTP/1.1" 200 None 2026-03-08 01:13:01,828 - lava_callback - INFO - Completed processing callback for node 69accd14bcd207e915d775de INFO: 172.21.0.1:60512 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:13:14,432 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:13:14,448 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:52560 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52564 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56298 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56310 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:13:33,354 - lava_callback - INFO - Received callback for node 69aa2a2b8c4d487a0e751eef INFO: 172.21.0.1:41022 - "POST /node/69aa2a2b8c4d487a0e751eef HTTP/1.0" 202 Accepted 2026-03-08 01:13:33,360 - lava_callback - INFO - Starting async job submit for node 69aa2a2b8c4d487a0e751eef 2026-03-08 01:13:33,361 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:13:33,375 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2a2b8c4d487a0e751eef HTTP/1.1" 200 None 2026-03-08 01:13:33,376 - lava_callback - DEBUG - Processing node 69aa2a2b8c4d487a0e751eef: baseline-arm64-baylibre 2026-03-08 01:13:33,376 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa2a2b8c4d487a0e751eef 2026-03-08 01:13:33,376 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:13:33,395 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2a2b8c4d487a0e751eef HTTP/1.1" 200 None 2026-03-08 01:13:33,396 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:13:33,409 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a70fafe31b554803095f11 HTTP/1.1" 200 None 2026-03-08 01:13:33,411 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:13:33,440 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa2a2b8c4d487a0e751eef HTTP/1.1" 200 None 2026-03-08 01:13:33,446 - lava_callback - ERROR - Error processing callback for node 69aa2a2b8c4d487a0e751eef: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:41032 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55514 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45884 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37234 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49634 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:49646 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34490 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34504 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50456 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41024 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39992 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45882 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41868 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:41884 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51678 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51692 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42314 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55176 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39172 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49560 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38086 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:38092 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35106 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35118 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49802 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50278 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46010 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39244 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35822 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:35834 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43534 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43544 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:17:33,527 - lava_callback - INFO - Received callback for node 69aa2b198c4d487a0e751f57 INFO: 172.21.0.1:42722 - "POST /node/69aa2b198c4d487a0e751f57 HTTP/1.0" 202 Accepted 2026-03-08 01:17:33,534 - lava_callback - INFO - Starting async job submit for node 69aa2b198c4d487a0e751f57 2026-03-08 01:17:33,535 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:17:33,549 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2b198c4d487a0e751f57 HTTP/1.1" 200 None 2026-03-08 01:17:33,550 - lava_callback - DEBUG - Processing node 69aa2b198c4d487a0e751f57: baseline-arm64-baylibre 2026-03-08 01:17:33,551 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa2b198c4d487a0e751f57 2026-03-08 01:17:33,551 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:17:33,567 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2b198c4d487a0e751f57 HTTP/1.1" 200 None 2026-03-08 01:17:33,568 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:17:33,583 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a70fafe31b554803095f11 HTTP/1.1" 200 None 2026-03-08 01:17:33,584 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:17:33,605 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa2b198c4d487a0e751f57 HTTP/1.1" 200 None 2026-03-08 01:17:33,606 - lava_callback - ERROR - Error processing callback for node 69aa2b198c4d487a0e751f57: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:42726 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42954 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39654 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33494 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47208 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47214 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33462 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33478 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36292 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36928 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:18:53,613 - lava_callback - INFO - Received callback for node 69aa2b698c4d487a0e751f88 INFO: 172.21.0.1:58334 - "POST /node/69aa2b698c4d487a0e751f88 HTTP/1.0" 202 Accepted 2026-03-08 01:18:53,619 - lava_callback - INFO - Starting async job submit for node 69aa2b698c4d487a0e751f88 2026-03-08 01:18:53,620 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:18:53,636 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2b698c4d487a0e751f88 HTTP/1.1" 200 None 2026-03-08 01:18:53,638 - lava_callback - DEBUG - Processing node 69aa2b698c4d487a0e751f88: baseline-arm64-baylibre 2026-03-08 01:18:53,638 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa2b698c4d487a0e751f88 2026-03-08 01:18:53,639 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:18:53,654 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2b698c4d487a0e751f88 HTTP/1.1" 200 None 2026-03-08 01:18:53,656 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:18:53,674 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a70fafe31b554803095f11 HTTP/1.1" 200 None 2026-03-08 01:18:53,676 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:18:53,715 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa2b698c4d487a0e751f88 HTTP/1.1" 200 None 2026-03-08 01:18:53,756 - lava_callback - ERROR - Error processing callback for node 69aa2b698c4d487a0e751f88: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:58338 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55502 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41858 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:41874 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57084 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57096 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:19:30,816 - lava_callback - INFO - Received callback for node 69acce87bcd207e915d776cc INFO: 172.21.0.1:33270 - "POST /node/69acce87bcd207e915d776cc HTTP/1.0" 202 Accepted 2026-03-08 01:19:30,824 - lava_callback - INFO - Starting async job submit for node 69acce87bcd207e915d776cc 2026-03-08 01:19:30,843 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:19:30,858 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce87bcd207e915d776cc HTTP/1.1" 200 None 2026-03-08 01:19:30,859 - lava_callback - DEBUG - Processing node 69acce87bcd207e915d776cc: baseline-arm64 2026-03-08 01:19:31,062 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:19:31,345 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:19:31,348 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce87bcd207e915d776cc/log.txt.gz 2026-03-08 01:19:31,484 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:19:31,671 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:19:31,672 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce87bcd207e915d776cc/lava-logs.yaml 2026-03-08 01:19:31,677 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:19:31,825 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:19:31,828 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce87bcd207e915d776cc/lava_callback.json.gz 2026-03-08 01:19:31,857 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:19:31,887 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce87bcd207e915d776cc HTTP/1.1" 200 None 2026-03-08 01:19:31,889 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:19:31,910 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:19:31,912 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:19:31,969 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce87bcd207e915d776cc HTTP/1.1" 200 None 2026-03-08 01:19:31,971 - lava_callback - INFO - Completed processing callback for node 69acce87bcd207e915d776cc INFO: 172.21.0.1:33272 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:19:44,451 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:19:44,478 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:38044 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59830 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:20:05,863 - lava_callback - INFO - Received callback for node 69accea3bcd207e915d77732 INFO: 172.21.0.1:60622 - "POST /node/69accea3bcd207e915d77732 HTTP/1.0" 202 Accepted 2026-03-08 01:20:05,869 - lava_callback - INFO - Starting async job submit for node 69accea3bcd207e915d77732 2026-03-08 01:20:05,871 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:05,893 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accea3bcd207e915d77732 HTTP/1.1" 200 None 2026-03-08 01:20:05,894 - lava_callback - DEBUG - Processing node 69accea3bcd207e915d77732: baseline-nfs-arm64 2026-03-08 01:20:05,895 - lava_callback - WARNING - Failed to upload log 2026-03-08 01:20:05,897 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:06,044 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:06,046 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea3bcd207e915d77732/lava-logs.yaml 2026-03-08 01:20:06,047 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:06,159 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:06,162 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea3bcd207e915d77732/lava_callback.json.gz 2026-03-08 01:20:06,164 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:06,178 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accea3bcd207e915d77732 HTTP/1.1" 200 None 2026-03-08 01:20:06,180 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:06,195 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:20:06,197 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:06,238 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accea3bcd207e915d77732 HTTP/1.1" 200 None 2026-03-08 01:20:06,241 - lava_callback - INFO - Completed processing callback for node 69accea3bcd207e915d77732 INFO: 172.21.0.1:60634 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:20:08,654 - lava_callback - INFO - Received callback for node 69acce93bcd207e915d77705 INFO: 172.21.0.1:60638 - "POST /node/69acce93bcd207e915d77705 HTTP/1.0" 202 Accepted 2026-03-08 01:20:08,661 - lava_callback - INFO - Starting async job submit for node 69acce93bcd207e915d77705 2026-03-08 01:20:08,680 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:08,694 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce93bcd207e915d77705 HTTP/1.1" 200 None 2026-03-08 01:20:08,695 - lava_callback - DEBUG - Processing node 69acce93bcd207e915d77705: baseline-arm64 2026-03-08 01:20:08,852 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:08,988 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:08,989 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce93bcd207e915d77705/log.txt.gz 2026-03-08 01:20:09,079 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:09,261 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:09,263 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce93bcd207e915d77705/lava-logs.yaml 2026-03-08 01:20:09,267 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:09,436 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:09,438 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce93bcd207e915d77705/lava_callback.json.gz 2026-03-08 01:20:09,463 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:09,520 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce93bcd207e915d77705 HTTP/1.1" 200 None 2026-03-08 01:20:09,523 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:09,582 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:20:09,585 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:09,648 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce93bcd207e915d77705 HTTP/1.1" 200 None 2026-03-08 01:20:09,650 - lava_callback - INFO - Completed processing callback for node 69acce93bcd207e915d77705 2026-03-08 01:20:14,058 - lava_callback - INFO - Received callback for node 69acce89bcd207e915d776d2 INFO: 172.21.0.1:39254 - "POST /node/69acce89bcd207e915d776d2 HTTP/1.0" 202 Accepted 2026-03-08 01:20:14,067 - lava_callback - INFO - Starting async job submit for node 69acce89bcd207e915d776d2 2026-03-08 01:20:14,100 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:14,155 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce89bcd207e915d776d2 HTTP/1.1" 200 None 2026-03-08 01:20:14,157 - lava_callback - DEBUG - Processing node 69acce89bcd207e915d776d2: baseline-arm64 2026-03-08 01:20:14,487 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:14,490 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:14,511 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-08 01:20:14,636 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:14,638 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce89bcd207e915d776d2/log.txt.gz 2026-03-08 01:20:14,817 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:15,116 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:15,118 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce89bcd207e915d776d2/lava-logs.yaml 2026-03-08 01:20:15,125 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:39260 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 01:20:15,281 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:15,283 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce89bcd207e915d776d2/lava_callback.json.gz 2026-03-08 01:20:15,322 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:15,346 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce89bcd207e915d776d2 HTTP/1.1" 200 None 2026-03-08 01:20:15,348 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:15,409 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:20:15,413 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:15,467 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce89bcd207e915d776d2 HTTP/1.1" 200 None 2026-03-08 01:20:15,469 - lava_callback - INFO - Completed processing callback for node 69acce89bcd207e915d776d2 INFO: 172.21.0.1:39272 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34366 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34368 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:20:35,770 - lava_callback - INFO - Received callback for node 69acce91bcd207e915d776ff INFO: 172.21.0.1:36152 - "POST /node/69acce91bcd207e915d776ff HTTP/1.0" 202 Accepted 2026-03-08 01:20:35,776 - lava_callback - INFO - Starting async job submit for node 69acce91bcd207e915d776ff 2026-03-08 01:20:35,799 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:35,814 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce91bcd207e915d776ff HTTP/1.1" 200 None 2026-03-08 01:20:35,815 - lava_callback - DEBUG - Processing node 69acce91bcd207e915d776ff: baseline-arm64 2026-03-08 01:20:36,055 - lava_callback - INFO - Received callback for node 69acce8bbcd207e915d776d9 INFO: 172.21.0.1:36168 - "POST /node/69acce8bbcd207e915d776d9 HTTP/1.0" 202 Accepted 2026-03-08 01:20:36,088 - lava_callback - INFO - Starting async job submit for node 69acce8bbcd207e915d776d9 2026-03-08 01:20:36,171 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:36,241 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce8bbcd207e915d776d9 HTTP/1.1" 200 None 2026-03-08 01:20:36,241 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:36,242 - lava_callback - DEBUG - Processing node 69acce8bbcd207e915d776d9: baseline-arm64 2026-03-08 01:20:36,412 - lava_callback - WARNING - Failed to upload log 2026-03-08 01:20:36,491 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:36,494 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:36,494 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce91bcd207e915d776ff/log.txt.gz INFO: 172.21.0.1:36182 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:20:36,753 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:36,766 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:36,766 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce8bbcd207e915d776d9/lava-logs.yaml 2026-03-08 01:20:36,771 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:36,967 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:36,969 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce8bbcd207e915d776d9/lava_callback.json.gz 2026-03-08 01:20:37,012 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:37,045 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce8bbcd207e915d776d9 HTTP/1.1" 200 None 2026-03-08 01:20:37,047 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:37,052 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:37,054 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce91bcd207e915d776ff/lava-logs.yaml 2026-03-08 01:20:37,066 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:37,066 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:20:37,068 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:37,112 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce8bbcd207e915d776d9 HTTP/1.1" 200 None 2026-03-08 01:20:37,181 - lava_callback - INFO - Completed processing callback for node 69acce8bbcd207e915d776d9 2026-03-08 01:20:37,182 - lava_callback - INFO - Received callback for node 69acce92bcd207e915d77702 INFO: 172.21.0.1:36196 - "POST /node/69acce92bcd207e915d77702 HTTP/1.0" 202 Accepted 2026-03-08 01:20:37,199 - lava_callback - INFO - Starting async job submit for node 69acce92bcd207e915d77702 2026-03-08 01:20:37,248 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:37,293 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce92bcd207e915d77702 HTTP/1.1" 200 None 2026-03-08 01:20:37,295 - lava_callback - DEBUG - Processing node 69acce92bcd207e915d77702: baseline-arm64 2026-03-08 01:20:37,368 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:37,381 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce91bcd207e915d776ff/lava_callback.json.gz 2026-03-08 01:20:37,445 - lava_callback - INFO - Received callback for node 69acce8cbcd207e915d776dd 2026-03-08 01:20:37,497 - lava_callback - INFO - Starting async job submit for node 69acce8cbcd207e915d776dd INFO: 172.21.0.1:36206 - "POST /node/69acce8cbcd207e915d776dd HTTP/1.0" 202 Accepted 2026-03-08 01:20:37,498 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:37,625 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce91bcd207e915d776ff HTTP/1.1" 200 None 2026-03-08 01:20:37,634 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:37,634 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:37,710 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:20:37,716 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce8cbcd207e915d776dd HTTP/1.1" 200 None 2026-03-08 01:20:37,728 - lava_callback - DEBUG - Processing node 69acce8cbcd207e915d776dd: baseline-arm64 2026-03-08 01:20:37,744 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:38,125 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce91bcd207e915d776ff HTTP/1.1" 200 None 2026-03-08 01:20:38,132 - lava_callback - INFO - Completed processing callback for node 69acce91bcd207e915d776ff 2026-03-08 01:20:38,133 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:38,173 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:38,449 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:38,451 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce8cbcd207e915d776dd/log.txt.gz 2026-03-08 01:20:38,465 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:38,499 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce92bcd207e915d77702/log.txt.gz 2026-03-08 01:20:38,600 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:38,851 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:38,854 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:38,854 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce8cbcd207e915d776dd/lava-logs.yaml 2026-03-08 01:20:38,859 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:38,990 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:38,992 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce8cbcd207e915d776dd/lava_callback.json.gz 2026-03-08 01:20:39,025 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:39,056 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce8cbcd207e915d776dd HTTP/1.1" 200 None 2026-03-08 01:20:39,059 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:39,076 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:20:39,078 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:39,104 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce8cbcd207e915d776dd HTTP/1.1" 200 None 2026-03-08 01:20:39,152 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:39,170 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce92bcd207e915d77702/lava-logs.yaml 2026-03-08 01:20:39,176 - lava_callback - INFO - Completed processing callback for node 69acce8cbcd207e915d776dd 2026-03-08 01:20:39,183 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:39,365 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:39,367 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce92bcd207e915d77702/lava_callback.json.gz 2026-03-08 01:20:39,390 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:39,414 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce92bcd207e915d77702 HTTP/1.1" 200 None 2026-03-08 01:20:39,415 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:39,429 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:20:39,430 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:39,453 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce92bcd207e915d77702 HTTP/1.1" 200 None 2026-03-08 01:20:39,459 - lava_callback - INFO - Completed processing callback for node 69acce92bcd207e915d77702 2026-03-08 01:20:44,513 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:44,540 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 2026-03-08 01:20:45,021 - lava_callback - INFO - Received callback for node 69acce90bcd207e915d776fc INFO: 172.21.0.1:59690 - "POST /node/69acce90bcd207e915d776fc HTTP/1.0" 202 Accepted 2026-03-08 01:20:45,028 - lava_callback - INFO - Starting async job submit for node 69acce90bcd207e915d776fc 2026-03-08 01:20:45,049 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:45,063 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce90bcd207e915d776fc HTTP/1.1" 200 None 2026-03-08 01:20:45,064 - lava_callback - DEBUG - Processing node 69acce90bcd207e915d776fc: baseline-arm64 2026-03-08 01:20:45,413 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:45,545 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:45,547 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce90bcd207e915d776fc/log.txt.gz 2026-03-08 01:20:45,758 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:45,965 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:45,967 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce90bcd207e915d776fc/lava-logs.yaml 2026-03-08 01:20:45,975 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:46,109 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:46,111 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce90bcd207e915d776fc/lava_callback.json.gz 2026-03-08 01:20:46,133 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:46,152 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce90bcd207e915d776fc HTTP/1.1" 200 None 2026-03-08 01:20:46,153 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:46,168 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:20:46,169 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:46,194 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce90bcd207e915d776fc HTTP/1.1" 200 None 2026-03-08 01:20:46,199 - lava_callback - INFO - Completed processing callback for node 69acce90bcd207e915d776fc 2026-03-08 01:20:46,485 - lava_callback - INFO - Received callback for node 69acce94bcd207e915d77708 INFO: 172.21.0.1:59704 - "POST /node/69acce94bcd207e915d77708 HTTP/1.0" 202 Accepted 2026-03-08 01:20:46,494 - lava_callback - INFO - Starting async job submit for node 69acce94bcd207e915d77708 2026-03-08 01:20:46,518 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:46,533 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce94bcd207e915d77708 HTTP/1.1" 200 None 2026-03-08 01:20:46,534 - lava_callback - DEBUG - Processing node 69acce94bcd207e915d77708: baseline-arm64 INFO: 172.21.0.1:59712 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:20:46,791 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:46,904 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:46,906 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce94bcd207e915d77708/log.txt.gz 2026-03-08 01:20:47,038 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:47,231 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:47,234 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce94bcd207e915d77708/lava-logs.yaml 2026-03-08 01:20:47,241 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:47,450 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:47,452 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce94bcd207e915d77708/lava_callback.json.gz 2026-03-08 01:20:47,475 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:47,490 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce94bcd207e915d77708 HTTP/1.1" 200 None 2026-03-08 01:20:47,492 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:47,508 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:20:47,510 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:47,541 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce94bcd207e915d77708 HTTP/1.1" 200 None 2026-03-08 01:20:47,543 - lava_callback - INFO - Completed processing callback for node 69acce94bcd207e915d77708 2026-03-08 01:20:54,865 - lava_callback - INFO - Received callback for node 69acc820bcd207e915d76efb INFO: 172.21.0.1:55404 - "POST /node/69acc820bcd207e915d76efb HTTP/1.0" 202 Accepted 2026-03-08 01:20:54,918 - lava_callback - INFO - Starting async job submit for node 69acc820bcd207e915d76efb 2026-03-08 01:20:54,964 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:54,988 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc820bcd207e915d76efb HTTP/1.1" 200 None 2026-03-08 01:20:54,990 - lava_callback - DEBUG - Processing node 69acc820bcd207e915d76efb: baseline-arm-pengutronix 2026-03-08 01:20:55,193 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:55,392 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:55,394 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69acc820bcd207e915d76efb/log.txt.gz 2026-03-08 01:20:55,459 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:55,614 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:55,616 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69acc820bcd207e915d76efb/lava-logs.yaml 2026-03-08 01:20:55,619 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:55,737 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:55,738 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69acc820bcd207e915d76efb/lava_callback.json.gz DEBUG: Setup failed for 69acc820bcd207e915d76efb. Transitting job node result: incomplete -> fail 2026-03-08 01:20:55,788 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:55,803 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc820bcd207e915d76efb HTTP/1.1" 200 None 2026-03-08 01:20:55,805 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:55,828 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 01:20:55,829 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:55,855 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc820bcd207e915d76efb HTTP/1.1" 200 None 2026-03-08 01:20:55,929 - lava_callback - INFO - Completed processing callback for node 69acc820bcd207e915d76efb 2026-03-08 01:20:56,045 - lava_callback - INFO - Received callback for node 69acce97bcd207e915d7770e INFO: 172.21.0.1:55408 - "POST /node/69acce97bcd207e915d7770e HTTP/1.0" 202 Accepted 2026-03-08 01:20:56,055 - lava_callback - INFO - Starting async job submit for node 69acce97bcd207e915d7770e 2026-03-08 01:20:56,085 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:56,102 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce97bcd207e915d7770e HTTP/1.1" 200 None 2026-03-08 01:20:56,103 - lava_callback - DEBUG - Processing node 69acce97bcd207e915d7770e: baseline-arm64 2026-03-08 01:20:56,397 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:56,520 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:56,522 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce97bcd207e915d7770e/log.txt.gz INFO: 172.21.0.1:55412 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:20:56,707 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:56,895 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:56,897 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce97bcd207e915d7770e/lava-logs.yaml 2026-03-08 01:20:56,904 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:57,030 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:57,032 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce97bcd207e915d7770e/lava_callback.json.gz 2026-03-08 01:20:57,055 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:57,069 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce97bcd207e915d7770e HTTP/1.1" 200 None 2026-03-08 01:20:57,071 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:57,085 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:20:57,086 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:57,118 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce97bcd207e915d7770e HTTP/1.1" 200 None 2026-03-08 01:20:57,120 - lava_callback - INFO - Completed processing callback for node 69acce97bcd207e915d7770e 2026-03-08 01:20:59,210 - lava_callback - INFO - Received callback for node 69acce88bcd207e915d776cf INFO: 172.21.0.1:55420 - "POST /node/69acce88bcd207e915d776cf HTTP/1.0" 202 Accepted 2026-03-08 01:20:59,218 - lava_callback - INFO - Starting async job submit for node 69acce88bcd207e915d776cf 2026-03-08 01:20:59,245 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:20:59,259 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce88bcd207e915d776cf HTTP/1.1" 200 None 2026-03-08 01:20:59,260 - lava_callback - DEBUG - Processing node 69acce88bcd207e915d776cf: baseline-arm64 2026-03-08 01:20:59,492 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:59,604 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:59,606 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce88bcd207e915d776cf/log.txt.gz 2026-03-08 01:20:59,763 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:20:59,941 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:20:59,943 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce88bcd207e915d776cf/lava-logs.yaml 2026-03-08 01:20:59,949 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:00,096 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:00,098 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce88bcd207e915d776cf/lava_callback.json.gz 2026-03-08 01:21:00,120 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:00,136 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce88bcd207e915d776cf HTTP/1.1" 200 None 2026-03-08 01:21:00,138 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:00,152 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:21:00,154 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:00,178 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce88bcd207e915d776cf HTTP/1.1" 200 None 2026-03-08 01:21:00,185 - lava_callback - INFO - Completed processing callback for node 69acce88bcd207e915d776cf 2026-03-08 01:21:01,631 - lava_callback - INFO - Received callback for node 69acce98bcd207e915d77711 INFO: 172.21.0.1:37476 - "POST /node/69acce98bcd207e915d77711 HTTP/1.0" 202 Accepted 2026-03-08 01:21:01,637 - lava_callback - INFO - Starting async job submit for node 69acce98bcd207e915d77711 2026-03-08 01:21:01,660 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:01,676 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce98bcd207e915d77711 HTTP/1.1" 200 None 2026-03-08 01:21:01,677 - lava_callback - DEBUG - Processing node 69acce98bcd207e915d77711: baseline-arm64 2026-03-08 01:21:01,932 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:02,049 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:02,051 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce98bcd207e915d77711/log.txt.gz 2026-03-08 01:21:02,210 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:02,390 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:02,392 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce98bcd207e915d77711/lava-logs.yaml 2026-03-08 01:21:02,398 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:02,527 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:02,528 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce98bcd207e915d77711/lava_callback.json.gz 2026-03-08 01:21:02,558 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:02,571 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce98bcd207e915d77711 HTTP/1.1" 200 None 2026-03-08 01:21:02,573 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:02,592 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:21:02,595 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:02,626 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce98bcd207e915d77711 HTTP/1.1" 200 None 2026-03-08 01:21:02,628 - lava_callback - INFO - Completed processing callback for node 69acce98bcd207e915d77711 INFO: 172.21.0.1:37492 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:21:07,002 - lava_callback - INFO - Received callback for node 69acce8ebcd207e915d776ea INFO: 172.21.0.1:37502 - "POST /node/69acce8ebcd207e915d776ea HTTP/1.0" 202 Accepted 2026-03-08 01:21:07,009 - lava_callback - INFO - Starting async job submit for node 69acce8ebcd207e915d776ea 2026-03-08 01:21:07,033 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:07,054 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce8ebcd207e915d776ea HTTP/1.1" 200 None 2026-03-08 01:21:07,055 - lava_callback - DEBUG - Processing node 69acce8ebcd207e915d776ea: baseline-arm64 2026-03-08 01:21:07,358 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:07,500 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:07,502 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce8ebcd207e915d776ea/log.txt.gz 2026-03-08 01:21:07,653 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:07,677 - lava_callback - INFO - Received callback for node 69accea0bcd207e915d7772b INFO: 172.21.0.1:37512 - "POST /node/69accea0bcd207e915d7772b HTTP/1.0" 202 Accepted 2026-03-08 01:21:07,685 - lava_callback - INFO - Starting async job submit for node 69accea0bcd207e915d7772b 2026-03-08 01:21:07,722 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:07,742 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accea0bcd207e915d7772b HTTP/1.1" 200 None 2026-03-08 01:21:07,743 - lava_callback - DEBUG - Processing node 69accea0bcd207e915d7772b: baseline-nfs-arm64 2026-03-08 01:21:07,883 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:07,921 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce8ebcd207e915d776ea/lava-logs.yaml 2026-03-08 01:21:08,025 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:08,233 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:08,243 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:08,245 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce8ebcd207e915d776ea/lava_callback.json.gz 2026-03-08 01:21:08,267 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:08,287 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce8ebcd207e915d776ea HTTP/1.1" 200 None 2026-03-08 01:21:08,289 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:08,315 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:21:08,317 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:08,352 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce8ebcd207e915d776ea HTTP/1.1" 200 None 2026-03-08 01:21:08,354 - lava_callback - INFO - Completed processing callback for node 69acce8ebcd207e915d776ea 2026-03-08 01:21:08,450 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:08,453 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea0bcd207e915d7772b/log.txt.gz 2026-03-08 01:21:08,787 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:08,999 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:09,001 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea0bcd207e915d7772b/lava-logs.yaml 2026-03-08 01:21:09,012 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:09,177 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:09,179 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea0bcd207e915d7772b/lava_callback.json.gz 2026-03-08 01:21:09,203 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:09,222 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accea0bcd207e915d7772b HTTP/1.1" 200 None 2026-03-08 01:21:09,224 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:09,238 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:21:09,240 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:09,263 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accea0bcd207e915d7772b HTTP/1.1" 200 None 2026-03-08 01:21:09,264 - lava_callback - INFO - Completed processing callback for node 69accea0bcd207e915d7772b 2026-03-08 01:21:09,992 - lava_callback - INFO - Received callback for node 69accea6bcd207e915d7773f INFO: 172.21.0.1:38202 - "POST /node/69accea6bcd207e915d7773f HTTP/1.0" 202 Accepted 2026-03-08 01:21:10,000 - lava_callback - INFO - Starting async job submit for node 69accea6bcd207e915d7773f 2026-03-08 01:21:10,025 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:10,050 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accea6bcd207e915d7773f HTTP/1.1" 200 None 2026-03-08 01:21:10,051 - lava_callback - DEBUG - Processing node 69accea6bcd207e915d7773f: baseline-nfs-arm64 2026-03-08 01:21:10,465 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:10,657 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:10,659 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea6bcd207e915d7773f/log.txt.gz 2026-03-08 01:21:10,899 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:11,177 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:11,180 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea6bcd207e915d7773f/lava-logs.yaml 2026-03-08 01:21:11,190 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:11,400 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:11,402 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea6bcd207e915d7773f/lava_callback.json.gz 2026-03-08 01:21:11,438 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:11,455 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accea6bcd207e915d7773f HTTP/1.1" 200 None 2026-03-08 01:21:11,457 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:11,473 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:21:11,475 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:11,516 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accea6bcd207e915d7773f HTTP/1.1" 200 None 2026-03-08 01:21:11,521 - lava_callback - INFO - Completed processing callback for node 69accea6bcd207e915d7773f 2026-03-08 01:21:12,222 - lava_callback - INFO - Received callback for node 69accea2bcd207e915d7772f INFO: 172.21.0.1:38204 - "POST /node/69accea2bcd207e915d7772f HTTP/1.0" 202 Accepted 2026-03-08 01:21:12,229 - lava_callback - INFO - Starting async job submit for node 69accea2bcd207e915d7772f 2026-03-08 01:21:12,262 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:12,281 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accea2bcd207e915d7772f HTTP/1.1" 200 None 2026-03-08 01:21:12,282 - lava_callback - DEBUG - Processing node 69accea2bcd207e915d7772f: baseline-nfs-arm64 2026-03-08 01:21:12,937 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:13,104 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:13,106 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea2bcd207e915d7772f/log.txt.gz 2026-03-08 01:21:13,489 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:13,694 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:13,697 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea2bcd207e915d7772f/lava-logs.yaml 2026-03-08 01:21:13,704 - lava_callback - INFO - Received callback for node 69aa2bf98c4d487a0e751fbc INFO: 172.21.0.1:38210 - "POST /node/69aa2bf98c4d487a0e751fbc HTTP/1.0" 202 Accepted 2026-03-08 01:21:13,711 - lava_callback - INFO - Starting async job submit for node 69aa2bf98c4d487a0e751fbc 2026-03-08 01:21:13,712 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:13,713 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:13,732 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2bf98c4d487a0e751fbc HTTP/1.1" 200 None 2026-03-08 01:21:13,733 - lava_callback - DEBUG - Processing node 69aa2bf98c4d487a0e751fbc: baseline-arm64-baylibre 2026-03-08 01:21:13,734 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa2bf98c4d487a0e751fbc 2026-03-08 01:21:13,734 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:13,762 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2bf98c4d487a0e751fbc HTTP/1.1" 200 None 2026-03-08 01:21:13,764 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:13,787 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a77a6ba255cc0324eb011a HTTP/1.1" 200 None 2026-03-08 01:21:13,789 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:13,819 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa2bf98c4d487a0e751fbc HTTP/1.1" 200 None 2026-03-08 01:21:13,820 - lava_callback - ERROR - Error processing callback for node 69aa2bf98c4d487a0e751fbc: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-08 01:21:13,947 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:13,949 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea2bcd207e915d7772f/lava_callback.json.gz 2026-03-08 01:21:13,985 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:14,005 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accea2bcd207e915d7772f HTTP/1.1" 200 None 2026-03-08 01:21:14,006 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:14,020 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:21:14,022 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:14,054 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accea2bcd207e915d7772f HTTP/1.1" 200 None 2026-03-08 01:21:14,055 - lava_callback - INFO - Completed processing callback for node 69accea2bcd207e915d7772f 2026-03-08 01:21:14,542 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:14,600 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 INFO: 172.21.0.1:38216 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:38218 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:21:16,669 - lava_callback - INFO - Received callback for node 69acce9ebcd207e915d77725 INFO: 172.21.0.1:38226 - "POST /node/69acce9ebcd207e915d77725 HTTP/1.0" 202 Accepted 2026-03-08 01:21:16,676 - lava_callback - INFO - Starting async job submit for node 69acce9ebcd207e915d77725 2026-03-08 01:21:16,698 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:16,718 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce9ebcd207e915d77725 HTTP/1.1" 200 None 2026-03-08 01:21:16,719 - lava_callback - DEBUG - Processing node 69acce9ebcd207e915d77725: baseline-nfs-arm64 2026-03-08 01:21:17,051 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:17,186 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:17,188 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acce9ebcd207e915d77725/log.txt.gz 2026-03-08 01:21:17,377 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:17,591 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:17,593 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acce9ebcd207e915d77725/lava-logs.yaml 2026-03-08 01:21:17,601 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:17,740 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:17,742 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acce9ebcd207e915d77725/lava_callback.json.gz 2026-03-08 01:21:17,787 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:17,809 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce9ebcd207e915d77725 HTTP/1.1" 200 None 2026-03-08 01:21:17,810 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:17,824 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:21:17,826 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:17,853 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce9ebcd207e915d77725 HTTP/1.1" 200 None 2026-03-08 01:21:17,855 - lava_callback - INFO - Completed processing callback for node 69acce9ebcd207e915d77725 2026-03-08 01:21:19,461 - lava_callback - INFO - Received callback for node 69acce99bcd207e915d77714 INFO: 172.21.0.1:38232 - "POST /node/69acce99bcd207e915d77714 HTTP/1.0" 202 Accepted 2026-03-08 01:21:19,467 - lava_callback - INFO - Starting async job submit for node 69acce99bcd207e915d77714 2026-03-08 01:21:19,488 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:19,509 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce99bcd207e915d77714 HTTP/1.1" 200 None 2026-03-08 01:21:19,509 - lava_callback - DEBUG - Processing node 69acce99bcd207e915d77714: baseline-arm64 2026-03-08 01:21:19,864 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:19,976 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:19,977 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce99bcd207e915d77714/log.txt.gz 2026-03-08 01:21:20,156 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:20,286 - lava_callback - INFO - Received callback for node 69acceaabcd207e915d77745 INFO: 172.21.0.1:34814 - "POST /node/69acceaabcd207e915d77745 HTTP/1.0" 202 Accepted 2026-03-08 01:21:20,294 - lava_callback - INFO - Starting async job submit for node 69acceaabcd207e915d77745 2026-03-08 01:21:20,318 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:20,342 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acceaabcd207e915d77745 HTTP/1.1" 200 None 2026-03-08 01:21:20,343 - lava_callback - DEBUG - Processing node 69acceaabcd207e915d77745: baseline-nfs-arm64 2026-03-08 01:21:20,362 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:20,414 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce99bcd207e915d77714/lava-logs.yaml 2026-03-08 01:21:20,504 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:34822 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:21:20,705 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:20,732 - lava_callback - INFO - Received callback for node 69accea4bcd207e915d77739 2026-03-08 01:21:20,737 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce99bcd207e915d77714/lava_callback.json.gz INFO: 172.21.0.1:34834 - "POST /node/69accea4bcd207e915d77739 HTTP/1.0" 202 Accepted 2026-03-08 01:21:20,759 - lava_callback - INFO - Starting async job submit for node 69accea4bcd207e915d77739 2026-03-08 01:21:20,772 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:20,813 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:20,813 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:20,826 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce99bcd207e915d77714 HTTP/1.1" 200 None 2026-03-08 01:21:20,828 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accea4bcd207e915d77739 HTTP/1.1" 200 None 2026-03-08 01:21:20,828 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:20,829 - lava_callback - DEBUG - Processing node 69accea4bcd207e915d77739: baseline-nfs-arm64 2026-03-08 01:21:20,919 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:21:20,941 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:21,068 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce99bcd207e915d77714 HTTP/1.1" 200 None 2026-03-08 01:21:21,070 - lava_callback - INFO - Completed processing callback for node 69acce99bcd207e915d77714 2026-03-08 01:21:21,160 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:21,182 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acceaabcd207e915d77745/log.txt.gz 2026-03-08 01:21:21,355 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:21,530 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:21,562 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:21,564 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea4bcd207e915d77739/log.txt.gz 2026-03-08 01:21:21,760 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:21,770 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:21,770 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acceaabcd207e915d77745/lava-logs.yaml 2026-03-08 01:21:21,780 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:21,935 - lava_callback - INFO - Received callback for node 69acce8abcd207e915d776d5 INFO: 172.21.0.1:34842 - "POST /node/69acce8abcd207e915d776d5 HTTP/1.0" 202 Accepted 2026-03-08 01:21:21,942 - lava_callback - INFO - Starting async job submit for node 69acce8abcd207e915d776d5 2026-03-08 01:21:21,964 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:21,966 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:21,968 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea4bcd207e915d77739/lava-logs.yaml 2026-03-08 01:21:21,976 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:21,982 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce8abcd207e915d776d5 HTTP/1.1" 200 None 2026-03-08 01:21:21,983 - lava_callback - DEBUG - Processing node 69acce8abcd207e915d776d5: baseline-arm64 2026-03-08 01:21:22,006 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:22,056 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acceaabcd207e915d77745/lava_callback.json.gz 2026-03-08 01:21:22,115 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:22,202 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acceaabcd207e915d77745 HTTP/1.1" 200 None 2026-03-08 01:21:22,203 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:22,228 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea4bcd207e915d77739/lava_callback.json.gz 2026-03-08 01:21:22,259 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:22,309 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:22,394 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accea4bcd207e915d77739 HTTP/1.1" 200 None 2026-03-08 01:21:22,405 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:21:22,413 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:22,421 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:22,449 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:22,469 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:21:22,472 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:22,499 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acceaabcd207e915d77745 HTTP/1.1" 200 None 2026-03-08 01:21:22,517 - lava_callback - INFO - Completed processing callback for node 69acceaabcd207e915d77745 2026-03-08 01:21:22,594 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accea4bcd207e915d77739 HTTP/1.1" 200 None 2026-03-08 01:21:22,596 - lava_callback - INFO - Completed processing callback for node 69accea4bcd207e915d77739 2026-03-08 01:21:22,706 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:22,708 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce8abcd207e915d776d5/log.txt.gz 2026-03-08 01:21:22,922 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:23,120 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:23,122 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce8abcd207e915d776d5/lava-logs.yaml 2026-03-08 01:21:23,129 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:23,275 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:23,277 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce8abcd207e915d776d5/lava_callback.json.gz 2026-03-08 01:21:23,300 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:23,314 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce8abcd207e915d776d5 HTTP/1.1" 200 None 2026-03-08 01:21:23,316 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:23,329 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:21:23,331 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:23,354 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce8abcd207e915d776d5 HTTP/1.1" 200 None 2026-03-08 01:21:23,357 - lava_callback - INFO - Completed processing callback for node 69acce8abcd207e915d776d5 INFO: 172.21.0.1:34858 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:21:27,837 - lava_callback - INFO - Received callback for node 69accea8bcd207e915d77742 INFO: 172.21.0.1:34864 - "POST /node/69accea8bcd207e915d77742 HTTP/1.0" 202 Accepted 2026-03-08 01:21:27,844 - lava_callback - INFO - Starting async job submit for node 69accea8bcd207e915d77742 2026-03-08 01:21:27,866 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:27,881 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accea8bcd207e915d77742 HTTP/1.1" 200 None 2026-03-08 01:21:27,882 - lava_callback - DEBUG - Processing node 69accea8bcd207e915d77742: baseline-nfs-arm64 2026-03-08 01:21:28,209 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:28,346 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:28,348 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea8bcd207e915d77742/log.txt.gz 2026-03-08 01:21:28,542 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:28,735 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:28,737 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea8bcd207e915d77742/lava-logs.yaml 2026-03-08 01:21:28,745 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:28,866 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:28,868 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea8bcd207e915d77742/lava_callback.json.gz 2026-03-08 01:21:28,891 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:28,906 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accea8bcd207e915d77742 HTTP/1.1" 200 None 2026-03-08 01:21:28,907 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:28,921 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:21:28,923 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:28,947 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accea8bcd207e915d77742 HTTP/1.1" 200 None 2026-03-08 01:21:28,949 - lava_callback - INFO - Completed processing callback for node 69accea8bcd207e915d77742 2026-03-08 01:21:30,563 - lava_callback - INFO - Received callback for node 69acce9bbcd207e915d7771e INFO: 172.21.0.1:48014 - "POST /node/69acce9bbcd207e915d7771e HTTP/1.0" 202 Accepted 2026-03-08 01:21:30,569 - lava_callback - INFO - Starting async job submit for node 69acce9bbcd207e915d7771e 2026-03-08 01:21:30,591 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:30,605 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce9bbcd207e915d7771e HTTP/1.1" 200 None 2026-03-08 01:21:30,606 - lava_callback - DEBUG - Processing node 69acce9bbcd207e915d7771e: baseline-nfs-arm64 2026-03-08 01:21:30,929 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:31,043 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:31,045 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acce9bbcd207e915d7771e/log.txt.gz 2026-03-08 01:21:31,236 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:31,421 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:31,423 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acce9bbcd207e915d7771e/lava-logs.yaml 2026-03-08 01:21:31,431 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:31,557 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:31,559 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acce9bbcd207e915d7771e/lava_callback.json.gz 2026-03-08 01:21:31,582 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:31,596 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce9bbcd207e915d7771e HTTP/1.1" 200 None 2026-03-08 01:21:31,598 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:31,611 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:21:31,612 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:31,645 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce9bbcd207e915d7771e HTTP/1.1" 200 None 2026-03-08 01:21:31,655 - lava_callback - INFO - Completed processing callback for node 69acce9bbcd207e915d7771e 2026-03-08 01:21:33,219 - lava_callback - INFO - Received callback for node 69acce95bcd207e915d7770b INFO: 172.21.0.1:48028 - "POST /node/69acce95bcd207e915d7770b HTTP/1.0" 202 Accepted 2026-03-08 01:21:33,225 - lava_callback - INFO - Starting async job submit for node 69acce95bcd207e915d7770b 2026-03-08 01:21:33,246 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:33,263 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce95bcd207e915d7770b HTTP/1.1" 200 None 2026-03-08 01:21:33,264 - lava_callback - DEBUG - Processing node 69acce95bcd207e915d7770b: baseline-arm64 2026-03-08 01:21:33,702 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:33,830 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:33,831 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce95bcd207e915d7770b/log.txt.gz 2026-03-08 01:21:34,033 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:34,249 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:34,251 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce95bcd207e915d7770b/lava-logs.yaml 2026-03-08 01:21:34,260 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:34,421 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:34,423 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce95bcd207e915d7770b/lava_callback.json.gz 2026-03-08 01:21:34,446 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:34,460 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce95bcd207e915d7770b HTTP/1.1" 200 None 2026-03-08 01:21:34,462 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:34,475 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:21:34,476 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:34,506 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce95bcd207e915d7770b HTTP/1.1" 200 None 2026-03-08 01:21:34,507 - lava_callback - INFO - Completed processing callback for node 69acce95bcd207e915d7770b INFO: 172.21.0.1:48032 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:21:39,169 - lava_callback - INFO - Received callback for node 69accecebcd207e915d777e3 INFO: 172.21.0.1:48034 - "POST /node/69accecebcd207e915d777e3 HTTP/1.0" 202 Accepted 2026-03-08 01:21:39,175 - lava_callback - INFO - Starting async job submit for node 69accecebcd207e915d777e3 2026-03-08 01:21:39,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:39,208 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accecebcd207e915d777e3 HTTP/1.1" 200 None 2026-03-08 01:21:39,209 - lava_callback - DEBUG - Processing node 69accecebcd207e915d777e3: baseline-arm64-pengutronix 2026-03-08 01:21:39,408 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:39,525 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:39,526 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69accecebcd207e915d777e3/log.txt.gz 2026-03-08 01:21:39,629 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:39,829 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:39,831 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69accecebcd207e915d777e3/lava-logs.yaml 2026-03-08 01:21:39,836 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:39,996 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:39,998 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69accecebcd207e915d777e3/lava_callback.json.gz 2026-03-08 01:21:40,018 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:40,033 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accecebcd207e915d777e3 HTTP/1.1" 200 None 2026-03-08 01:21:40,034 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:40,058 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:21:40,059 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:40,084 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accecebcd207e915d777e3 HTTP/1.1" 200 None 2026-03-08 01:21:40,094 - lava_callback - INFO - Completed processing callback for node 69accecebcd207e915d777e3 2026-03-08 01:21:44,603 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:44,621 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 15 INFO: 172.21.0.1:33752 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:21:52,873 - lava_callback - INFO - Received callback for node 69acced0bcd207e915d777f1 INFO: 172.21.0.1:48058 - "POST /node/69acced0bcd207e915d777f1 HTTP/1.0" 202 Accepted 2026-03-08 01:21:52,879 - lava_callback - INFO - Starting async job submit for node 69acced0bcd207e915d777f1 2026-03-08 01:21:52,898 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:52,912 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acced0bcd207e915d777f1 HTTP/1.1" 200 None 2026-03-08 01:21:52,913 - lava_callback - DEBUG - Processing node 69acced0bcd207e915d777f1: baseline-arm64-pengutronix 2026-03-08 01:21:53,118 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:53,238 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:53,240 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69acced0bcd207e915d777f1/log.txt.gz 2026-03-08 01:21:53,362 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:53,563 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:53,565 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69acced0bcd207e915d777f1/lava-logs.yaml 2026-03-08 01:21:53,571 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:21:53,713 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:21:53,715 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69acced0bcd207e915d777f1/lava_callback.json.gz 2026-03-08 01:21:53,735 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:53,749 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acced0bcd207e915d777f1 HTTP/1.1" 200 None 2026-03-08 01:21:53,750 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:53,764 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:21:53,765 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:21:53,792 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acced0bcd207e915d777f1 HTTP/1.1" 200 None 2026-03-08 01:21:53,797 - lava_callback - INFO - Completed processing callback for node 69acced0bcd207e915d777f1 INFO: 172.21.0.1:48072 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:22:02,927 - lava_callback - INFO - Received callback for node 69accea5bcd207e915d7773c INFO: 172.21.0.1:51748 - "POST /node/69accea5bcd207e915d7773c HTTP/1.0" 202 Accepted 2026-03-08 01:22:02,933 - lava_callback - INFO - Starting async job submit for node 69accea5bcd207e915d7773c 2026-03-08 01:22:02,955 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:02,979 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accea5bcd207e915d7773c HTTP/1.1" 200 None 2026-03-08 01:22:02,979 - lava_callback - DEBUG - Processing node 69accea5bcd207e915d7773c: baseline-nfs-arm64 2026-03-08 01:22:03,432 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:22:03,569 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:22:03,571 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea5bcd207e915d7773c/log.txt.gz 2026-03-08 01:22:03,813 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:22:04,009 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:22:04,011 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea5bcd207e915d7773c/lava-logs.yaml 2026-03-08 01:22:04,022 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:22:04,253 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:22:04,255 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69accea5bcd207e915d7773c/lava_callback.json.gz 2026-03-08 01:22:04,277 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:04,291 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accea5bcd207e915d7773c HTTP/1.1" 200 None 2026-03-08 01:22:04,292 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:04,306 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:22:04,308 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:04,339 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accea5bcd207e915d7773c HTTP/1.1" 200 None 2026-03-08 01:22:04,340 - lava_callback - INFO - Completed processing callback for node 69accea5bcd207e915d7773c 2026-03-08 01:22:05,420 - lava_callback - INFO - Received callback for node 69accec7bcd207e915d777ac INFO: 172.21.0.1:51764 - "POST /node/69accec7bcd207e915d777ac HTTP/1.0" 202 Accepted 2026-03-08 01:22:05,426 - lava_callback - INFO - Starting async job submit for node 69accec7bcd207e915d777ac 2026-03-08 01:22:05,447 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:05,461 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accec7bcd207e915d777ac HTTP/1.1" 200 None 2026-03-08 01:22:05,462 - lava_callback - DEBUG - Processing node 69accec7bcd207e915d777ac: baseline-arm64-clabbe 2026-03-08 01:22:05,673 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:22:05,786 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:22:05,788 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69accec7bcd207e915d777ac/log.txt.gz 2026-03-08 01:22:05,914 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:22:06,102 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:22:06,104 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69accec7bcd207e915d777ac/lava-logs.yaml 2026-03-08 01:22:06,109 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:22:06,241 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:22:06,243 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69accec7bcd207e915d777ac/lava_callback.json.gz 2026-03-08 01:22:06,266 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:06,279 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accec7bcd207e915d777ac HTTP/1.1" 200 None 2026-03-08 01:22:06,281 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:06,293 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:22:06,295 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:06,319 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accec7bcd207e915d777ac HTTP/1.1" 200 None 2026-03-08 01:22:06,321 - lava_callback - INFO - Completed processing callback for node 69accec7bcd207e915d777ac INFO: 172.21.0.1:51770 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:22:14,623 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:14,639 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:35520 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:35526 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:22:19,983 - lava_callback - INFO - Received callback for node 69accec9bcd207e915d777b5 INFO: 172.21.0.1:59514 - "POST /node/69accec9bcd207e915d777b5 HTTP/1.0" 202 Accepted 2026-03-08 01:22:19,993 - lava_callback - INFO - Starting async job submit for node 69accec9bcd207e915d777b5 2026-03-08 01:22:20,028 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:20,042 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accec9bcd207e915d777b5 HTTP/1.1" 200 None 2026-03-08 01:22:20,043 - lava_callback - DEBUG - Processing node 69accec9bcd207e915d777b5: baseline-arm64-clabbe 2026-03-08 01:22:20,264 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:22:20,391 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:22:20,393 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69accec9bcd207e915d777b5/log.txt.gz 2026-03-08 01:22:20,545 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:59524 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:22:20,739 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:22:20,741 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69accec9bcd207e915d777b5/lava-logs.yaml 2026-03-08 01:22:20,747 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:22:20,906 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:22:20,908 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69accec9bcd207e915d777b5/lava_callback.json.gz 2026-03-08 01:22:20,930 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:20,945 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accec9bcd207e915d777b5 HTTP/1.1" 200 None 2026-03-08 01:22:20,946 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:20,960 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:22:20,962 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:20,986 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accec9bcd207e915d777b5 HTTP/1.1" 200 None 2026-03-08 01:22:20,988 - lava_callback - INFO - Completed processing callback for node 69accec9bcd207e915d777b5 2026-03-08 01:22:21,372 - lava_callback - INFO - Received callback for node 69accec5bcd207e915d777a7 INFO: 172.21.0.1:59534 - "POST /node/69accec5bcd207e915d777a7 HTTP/1.0" 202 Accepted 2026-03-08 01:22:21,379 - lava_callback - INFO - Starting async job submit for node 69accec5bcd207e915d777a7 2026-03-08 01:22:21,400 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:21,415 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accec5bcd207e915d777a7 HTTP/1.1" 200 None 2026-03-08 01:22:21,415 - lava_callback - DEBUG - Processing node 69accec5bcd207e915d777a7: baseline-arm64-clabbe 2026-03-08 01:22:21,729 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:22:21,860 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:22:21,862 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69accec5bcd207e915d777a7/log.txt.gz 2026-03-08 01:22:22,045 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:22:22,228 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:22:22,231 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69accec5bcd207e915d777a7/lava-logs.yaml 2026-03-08 01:22:22,238 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:22:22,378 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:22:22,380 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69accec5bcd207e915d777a7/lava_callback.json.gz 2026-03-08 01:22:22,403 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:22,426 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accec5bcd207e915d777a7 HTTP/1.1" 200 None 2026-03-08 01:22:22,427 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:22,441 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:22:22,443 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:22,467 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accec5bcd207e915d777a7 HTTP/1.1" 200 None 2026-03-08 01:22:22,468 - lava_callback - INFO - Completed processing callback for node 69accec5bcd207e915d777a7 INFO: 172.21.0.1:59540 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:22:33,811 - lava_callback - INFO - Received callback for node 69aa2c498c4d487a0e751fd8 INFO: 172.21.0.1:56676 - "POST /node/69aa2c498c4d487a0e751fd8 HTTP/1.0" 202 Accepted 2026-03-08 01:22:33,817 - lava_callback - INFO - Starting async job submit for node 69aa2c498c4d487a0e751fd8 2026-03-08 01:22:33,818 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:33,842 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2c498c4d487a0e751fd8 HTTP/1.1" 200 None 2026-03-08 01:22:33,843 - lava_callback - DEBUG - Processing node 69aa2c498c4d487a0e751fd8: baseline-arm64-baylibre 2026-03-08 01:22:33,844 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa2c498c4d487a0e751fd8 2026-03-08 01:22:33,844 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:33,862 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2c498c4d487a0e751fd8 HTTP/1.1" 200 None 2026-03-08 01:22:33,863 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:33,876 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a77a6ba255cc0324eb011a HTTP/1.1" 200 None 2026-03-08 01:22:33,878 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:33,898 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa2c498c4d487a0e751fd8 HTTP/1.1" 200 None 2026-03-08 01:22:33,899 - lava_callback - ERROR - Error processing callback for node 69aa2c498c4d487a0e751fd8: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:56678 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:22:44,641 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:44,658 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-08 01:22:45,626 - lava_callback - INFO - Received callback for node 69accecfbcd207e915d777ea INFO: 172.21.0.1:39450 - "POST /node/69accecfbcd207e915d777ea HTTP/1.0" 202 Accepted 2026-03-08 01:22:45,632 - lava_callback - INFO - Starting async job submit for node 69accecfbcd207e915d777ea 2026-03-08 01:22:45,652 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:45,667 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accecfbcd207e915d777ea HTTP/1.1" 200 None 2026-03-08 01:22:45,667 - lava_callback - DEBUG - Processing node 69accecfbcd207e915d777ea: baseline-arm64-pengutronix 2026-03-08 01:22:45,859 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:22:45,977 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:22:45,978 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69accecfbcd207e915d777ea/log.txt.gz 2026-03-08 01:22:46,098 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:22:46,287 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:22:46,289 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69accecfbcd207e915d777ea/lava-logs.yaml 2026-03-08 01:22:46,294 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:22:46,454 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:22:46,455 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69accecfbcd207e915d777ea/lava_callback.json.gz 2026-03-08 01:22:46,502 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:46,516 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accecfbcd207e915d777ea HTTP/1.1" 200 None 2026-03-08 01:22:46,518 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.21.0.1:39456 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:22:46,543 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:22:46,545 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:22:46,574 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accecfbcd207e915d777ea HTTP/1.1" 200 None 2026-03-08 01:22:46,636 - lava_callback - INFO - Completed processing callback for node 69accecfbcd207e915d777ea INFO: 172.21.0.1:53062 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44090 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:23:14,660 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:23:14,675 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:56784 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56800 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:23:19,864 - lava_callback - INFO - Received callback for node 69acce8dbcd207e915d776e7 INFO: 172.21.0.1:55024 - "POST /node/69acce8dbcd207e915d776e7 HTTP/1.0" 202 Accepted 2026-03-08 01:23:19,870 - lava_callback - INFO - Starting async job submit for node 69acce8dbcd207e915d776e7 2026-03-08 01:23:19,887 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:23:19,902 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce8dbcd207e915d776e7 HTTP/1.1" 200 None 2026-03-08 01:23:19,902 - lava_callback - DEBUG - Processing node 69acce8dbcd207e915d776e7: baseline-arm64 2026-03-08 01:23:20,293 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:23:20,435 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:23:20,437 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce8dbcd207e915d776e7/log.txt.gz INFO: 172.21.0.1:55032 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:23:20,627 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:23:20,849 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:23:20,852 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce8dbcd207e915d776e7/lava-logs.yaml 2026-03-08 01:23:20,861 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:23:21,007 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:23:21,008 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce8dbcd207e915d776e7/lava_callback.json.gz 2026-03-08 01:23:21,026 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:23:21,040 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce8dbcd207e915d776e7 HTTP/1.1" 200 None 2026-03-08 01:23:21,042 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:23:21,064 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:23:21,065 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:23:21,099 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce8dbcd207e915d776e7 HTTP/1.1" 200 None 2026-03-08 01:23:21,101 - lava_callback - INFO - Completed processing callback for node 69acce8dbcd207e915d776e7 INFO: 172.21.0.1:55036 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:23:35,278 - lava_callback - INFO - Received callback for node 69accecbbcd207e915d777c0 INFO: 172.21.0.1:37000 - "POST /node/69accecbbcd207e915d777c0 HTTP/1.0" 202 Accepted 2026-03-08 01:23:35,289 - lava_callback - INFO - Starting async job submit for node 69accecbbcd207e915d777c0 2026-03-08 01:23:35,313 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:23:35,326 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accecbbcd207e915d777c0 HTTP/1.1" 200 None 2026-03-08 01:23:35,327 - lava_callback - DEBUG - Processing node 69accecbbcd207e915d777c0: baseline-arm64-clabbe 2026-03-08 01:23:35,557 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:23:35,735 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:23:35,737 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69accecbbcd207e915d777c0/log.txt.gz 2026-03-08 01:23:35,886 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:23:36,083 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:23:36,085 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69accecbbcd207e915d777c0/lava-logs.yaml 2026-03-08 01:23:36,091 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:23:36,226 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:23:36,229 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69accecbbcd207e915d777c0/lava_callback.json.gz 2026-03-08 01:23:36,256 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:23:36,271 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accecbbcd207e915d777c0 HTTP/1.1" 200 None 2026-03-08 01:23:36,273 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:23:36,286 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:23:36,288 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:23:36,312 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accecbbcd207e915d777c0 HTTP/1.1" 200 None 2026-03-08 01:23:36,313 - lava_callback - INFO - Completed processing callback for node 69accecbbcd207e915d777c0 INFO: 172.21.0.1:37012 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:23:44,677 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:23:44,693 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:46526 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:23:53,853 - lava_callback - INFO - Received callback for node 69aa2c9b8c4d487a0e751ff9 INFO: 172.21.0.1:49020 - "POST /node/69aa2c9b8c4d487a0e751ff9 HTTP/1.0" 202 Accepted 2026-03-08 01:23:53,859 - lava_callback - INFO - Starting async job submit for node 69aa2c9b8c4d487a0e751ff9 2026-03-08 01:23:53,860 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:23:53,874 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2c9b8c4d487a0e751ff9 HTTP/1.1" 200 None 2026-03-08 01:23:53,875 - lava_callback - DEBUG - Processing node 69aa2c9b8c4d487a0e751ff9: baseline-arm64-baylibre 2026-03-08 01:23:53,875 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa2c9b8c4d487a0e751ff9 2026-03-08 01:23:53,876 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:23:53,895 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2c9b8c4d487a0e751ff9 HTTP/1.1" 200 None 2026-03-08 01:23:53,896 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:23:53,909 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a77a6ba255cc0324eb011a HTTP/1.1" 200 None 2026-03-08 01:23:53,911 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:23:53,931 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa2c9b8c4d487a0e751ff9 HTTP/1.1" 200 None 2026-03-08 01:23:53,935 - lava_callback - ERROR - Error processing callback for node 69aa2c9b8c4d487a0e751ff9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:49024 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:23:59,777 - lava_callback - INFO - Received callback for node 69accec8bcd207e915d777af INFO: 172.21.0.1:46210 - "POST /node/69accec8bcd207e915d777af HTTP/1.0" 202 Accepted 2026-03-08 01:23:59,784 - lava_callback - INFO - Starting async job submit for node 69accec8bcd207e915d777af 2026-03-08 01:23:59,808 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:23:59,822 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accec8bcd207e915d777af HTTP/1.1" 200 None 2026-03-08 01:23:59,823 - lava_callback - DEBUG - Processing node 69accec8bcd207e915d777af: baseline-arm64-clabbe 2026-03-08 01:24:00,195 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:24:00,351 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:24:00,352 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69accec8bcd207e915d777af/log.txt.gz 2026-03-08 01:24:00,544 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:24:00,804 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:24:00,806 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69accec8bcd207e915d777af/lava-logs.yaml 2026-03-08 01:24:00,817 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:24:00,992 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:24:00,994 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-clabbe-69accec8bcd207e915d777af/lava_callback.json.gz 2026-03-08 01:24:01,023 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:24:01,041 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accec8bcd207e915d777af HTTP/1.1" 200 None 2026-03-08 01:24:01,043 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:24:01,056 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:24:01,058 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:24:01,087 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accec8bcd207e915d777af HTTP/1.1" 200 None 2026-03-08 01:24:01,088 - lava_callback - INFO - Completed processing callback for node 69accec8bcd207e915d777af INFO: 172.21.0.1:46220 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:24:14,695 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:24:14,710 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:32796 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:32810 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36270 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:24:24,042 - lava_callback - INFO - Received callback for node 69acce9dbcd207e915d77722 INFO: 172.21.0.1:36276 - "POST /node/69acce9dbcd207e915d77722 HTTP/1.0" 202 Accepted 2026-03-08 01:24:24,049 - lava_callback - INFO - Starting async job submit for node 69acce9dbcd207e915d77722 2026-03-08 01:24:24,068 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:24:24,082 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce9dbcd207e915d77722 HTTP/1.1" 200 None 2026-03-08 01:24:24,083 - lava_callback - DEBUG - Processing node 69acce9dbcd207e915d77722: baseline-nfs-arm64 2026-03-08 01:24:24,465 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:24:24,652 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:24:24,654 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acce9dbcd207e915d77722/log.txt.gz 2026-03-08 01:24:24,900 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:24:25,128 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:24:25,130 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acce9dbcd207e915d77722/lava-logs.yaml 2026-03-08 01:24:25,140 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:24:25,306 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:24:25,308 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acce9dbcd207e915d77722/lava_callback.json.gz 2026-03-08 01:24:25,332 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:24:25,346 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce9dbcd207e915d77722 HTTP/1.1" 200 None 2026-03-08 01:24:25,348 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:24:25,366 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:24:25,367 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:24:25,395 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce9dbcd207e915d77722 HTTP/1.1" 200 None 2026-03-08 01:24:25,406 - lava_callback - INFO - Completed processing callback for node 69acce9dbcd207e915d77722 INFO: 172.21.0.1:36278 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39792 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:24:44,712 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:24:44,728 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:54102 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58046 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50656 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:25:13,903 - lava_callback - INFO - Received callback for node 69aa2cec8c4d487a0e75201a INFO: 172.21.0.1:55022 - "POST /node/69aa2cec8c4d487a0e75201a HTTP/1.0" 202 Accepted 2026-03-08 01:25:13,909 - lava_callback - INFO - Starting async job submit for node 69aa2cec8c4d487a0e75201a 2026-03-08 01:25:13,910 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:25:13,927 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2cec8c4d487a0e75201a HTTP/1.1" 200 None 2026-03-08 01:25:13,928 - lava_callback - DEBUG - Processing node 69aa2cec8c4d487a0e75201a: baseline-arm64-baylibre 2026-03-08 01:25:13,929 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa2cec8c4d487a0e75201a 2026-03-08 01:25:13,929 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:25:13,942 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2cec8c4d487a0e75201a HTTP/1.1" 200 None 2026-03-08 01:25:13,944 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:25:13,957 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a77a6ba255cc0324eb011a HTTP/1.1" 200 None 2026-03-08 01:25:13,958 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:25:13,984 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa2cec8c4d487a0e75201a HTTP/1.1" 200 None 2026-03-08 01:25:13,986 - lava_callback - ERROR - Error processing callback for node 69aa2cec8c4d487a0e75201a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:55038 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55048 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58086 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58100 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:25:33,446 - lava_callback - INFO - Received callback for node 69accbd9bcd207e915d77547 INFO: 172.21.0.1:37714 - "POST /node/69accbd9bcd207e915d77547 HTTP/1.0" 202 Accepted 2026-03-08 01:25:33,452 - lava_callback - INFO - Starting async job submit for node 69accbd9bcd207e915d77547 2026-03-08 01:25:33,509 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:25:33,523 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accbd9bcd207e915d77547 HTTP/1.1" 200 None 2026-03-08 01:25:33,524 - lava_callback - DEBUG - Processing node 69accbd9bcd207e915d77547: ltp-smoke 2026-03-08 01:25:33,994 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:25:34,150 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:25:34,152 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69accbd9bcd207e915d77547/log.txt.gz 2026-03-08 01:25:34,444 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:25:34,680 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:25:34,682 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69accbd9bcd207e915d77547/lava-logs.yaml 2026-03-08 01:25:34,694 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:25:34,861 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:25:34,863 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69accbd9bcd207e915d77547/lava_callback.json.gz 2026-03-08 01:25:34,999 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:25:35,026 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accbd9bcd207e915d77547 HTTP/1.1" 200 None 2026-03-08 01:25:35,027 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:25:35,041 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc012bcd207e915d76b9c HTTP/1.1" 200 None 2026-03-08 01:25:35,042 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:25:35,068 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accbd9bcd207e915d77547 HTTP/1.1" 200 None 2026-03-08 01:25:35,198 - lava_callback - INFO - Completed processing callback for node 69accbd9bcd207e915d77547 INFO: 172.21.0.1:37716 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:25:44,730 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:25:44,748 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:59600 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52596 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60722 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40396 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:40408 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33734 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33736 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:26:34,030 - lava_callback - INFO - Received callback for node 69aa2d3d8c4d487a0e75204a INFO: 172.21.0.1:34946 - "POST /node/69aa2d3d8c4d487a0e75204a HTTP/1.0" 202 Accepted 2026-03-08 01:26:34,036 - lava_callback - INFO - Starting async job submit for node 69aa2d3d8c4d487a0e75204a 2026-03-08 01:26:34,037 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:26:34,050 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2d3d8c4d487a0e75204a HTTP/1.1" 200 None 2026-03-08 01:26:34,051 - lava_callback - DEBUG - Processing node 69aa2d3d8c4d487a0e75204a: baseline-arm64-baylibre 2026-03-08 01:26:34,051 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa2d3d8c4d487a0e75204a 2026-03-08 01:26:34,052 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:26:34,065 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2d3d8c4d487a0e75204a HTTP/1.1" 200 None 2026-03-08 01:26:34,066 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:26:34,084 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a77a6ba255cc0324eb011a HTTP/1.1" 200 None 2026-03-08 01:26:34,085 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:26:34,105 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa2d3d8c4d487a0e75204a HTTP/1.1" 200 None 2026-03-08 01:26:34,107 - lava_callback - ERROR - Error processing callback for node 69aa2d3d8c4d487a0e75204a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:34956 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46544 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48084 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:27:06,109 - lava_callback - INFO - Received callback for node 69acd04bbcd207e915d77963 INFO: 172.21.0.1:60670 - "POST /node/69acd04bbcd207e915d77963 HTTP/1.0" 202 Accepted 2026-03-08 01:27:06,115 - lava_callback - INFO - Starting async job submit for node 69acd04bbcd207e915d77963 2026-03-08 01:27:06,117 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:27:06,131 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd04bbcd207e915d77963 HTTP/1.1" 200 None 2026-03-08 01:27:06,132 - lava_callback - DEBUG - Processing node 69acd04bbcd207e915d77963: baseline-nfs-arm64 2026-03-08 01:27:06,133 - lava_callback - WARNING - Failed to upload log 2026-03-08 01:27:06,135 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:27:06,298 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:27:06,299 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acd04bbcd207e915d77963/lava-logs.yaml 2026-03-08 01:27:06,301 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:27:06,445 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:27:06,446 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acd04bbcd207e915d77963/lava_callback.json.gz 2026-03-08 01:27:06,448 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:27:06,462 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd04bbcd207e915d77963 HTTP/1.1" 200 None 2026-03-08 01:27:06,464 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:27:06,478 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:27:06,480 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:27:06,504 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd04bbcd207e915d77963 HTTP/1.1" 200 None 2026-03-08 01:27:06,507 - lava_callback - INFO - Completed processing callback for node 69acd04bbcd207e915d77963 INFO: 172.21.0.1:60682 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:27:14,750 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:27:14,766 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:38826 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:38840 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:27:20,595 - lava_callback - INFO - Received callback for node 69acce9abcd207e915d7771a INFO: 172.21.0.1:60046 - "POST /node/69acce9abcd207e915d7771a HTTP/1.0" 202 Accepted 2026-03-08 01:27:20,602 - lava_callback - INFO - Starting async job submit for node 69acce9abcd207e915d7771a 2026-03-08 01:27:20,637 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.21.0.1:60054 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:27:20,653 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce9abcd207e915d7771a HTTP/1.1" 200 None 2026-03-08 01:27:20,653 - lava_callback - DEBUG - Processing node 69acce9abcd207e915d7771a: baseline-arm64 2026-03-08 01:27:21,024 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:27:21,173 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:27:21,175 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce9abcd207e915d7771a/log.txt.gz 2026-03-08 01:27:21,382 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:27:21,591 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:27:21,593 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce9abcd207e915d7771a/lava-logs.yaml 2026-03-08 01:27:21,602 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:27:21,767 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:27:21,768 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acce9abcd207e915d7771a/lava_callback.json.gz DEBUG: Setup failed for 69acce9abcd207e915d7771a. Transitting job node result: incomplete -> fail 2026-03-08 01:27:21,840 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:27:21,860 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce9abcd207e915d7771a HTTP/1.1" 200 None 2026-03-08 01:27:21,861 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:27:21,876 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:27:21,878 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:27:21,916 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce9abcd207e915d7771a HTTP/1.1" 200 None 2026-03-08 01:27:22,004 - lava_callback - INFO - Completed processing callback for node 69acce9abcd207e915d7771a INFO: 172.21.0.1:60062 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49146 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:27:44,769 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:27:44,810 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:60268 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:27:54,195 - lava_callback - INFO - Received callback for node 69aa2d8c8c4d487a0e75206a INFO: 172.21.0.1:56366 - "POST /node/69aa2d8c8c4d487a0e75206a HTTP/1.0" 202 Accepted 2026-03-08 01:27:54,200 - lava_callback - INFO - Starting async job submit for node 69aa2d8c8c4d487a0e75206a 2026-03-08 01:27:54,201 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:27:54,216 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2d8c8c4d487a0e75206a HTTP/1.1" 200 None 2026-03-08 01:27:54,216 - lava_callback - DEBUG - Processing node 69aa2d8c8c4d487a0e75206a: baseline-arm64-baylibre 2026-03-08 01:27:54,217 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa2d8c8c4d487a0e75206a 2026-03-08 01:27:54,217 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:27:54,231 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2d8c8c4d487a0e75206a HTTP/1.1" 200 None 2026-03-08 01:27:54,232 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:27:54,246 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a77a6ba255cc0324eb011a HTTP/1.1" 200 None 2026-03-08 01:27:54,247 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:27:54,295 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa2d8c8c4d487a0e75206a HTTP/1.1" 200 None 2026-03-08 01:27:54,297 - lava_callback - ERROR - Error processing callback for node 69aa2d8c8c4d487a0e75206a: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:56368 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56192 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42562 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:42570 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58310 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58318 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48762 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:28:41,799 - lava_callback - INFO - Received callback for node 69accd10bcd207e915d775db INFO: 172.21.0.1:45876 - "POST /node/69accd10bcd207e915d775db HTTP/1.0" 202 Accepted 2026-03-08 01:28:41,806 - lava_callback - INFO - Starting async job submit for node 69accd10bcd207e915d775db 2026-03-08 01:28:41,854 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:28:41,869 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accd10bcd207e915d775db HTTP/1.1" 200 None 2026-03-08 01:28:41,869 - lava_callback - DEBUG - Processing node 69accd10bcd207e915d775db: baseline-x86 2026-03-08 01:28:42,661 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:28:42,859 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:28:42,861 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69accd10bcd207e915d775db/log.txt.gz 2026-03-08 01:28:43,305 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:28:43,555 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:28:43,558 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69accd10bcd207e915d775db/lava-logs.yaml 2026-03-08 01:28:43,578 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:28:43,783 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:28:43,785 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69accd10bcd207e915d775db/lava_callback.json.gz DEBUG: Setup failed for 69accd10bcd207e915d775db. Transitting job node result: incomplete -> fail 2026-03-08 01:28:43,873 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:28:43,893 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accd10bcd207e915d775db HTTP/1.1" 200 None 2026-03-08 01:28:43,895 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:28:43,914 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc037bcd207e915d76bc8 HTTP/1.1" 200 None 2026-03-08 01:28:43,916 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:28:43,941 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accd10bcd207e915d775db HTTP/1.1" 200 None 2026-03-08 01:28:44,033 - lava_callback - INFO - Completed processing callback for node 69accd10bcd207e915d775db 2026-03-08 01:28:44,812 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:28:44,829 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:45886 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:28:47,840 - lava_callback - INFO - Received callback for node 69acd093bcd207e915d77990 INFO: 172.21.0.1:45894 - "POST /node/69acd093bcd207e915d77990 HTTP/1.0" 202 Accepted 2026-03-08 01:28:47,848 - lava_callback - INFO - Starting async job submit for node 69acd093bcd207e915d77990 2026-03-08 01:28:47,886 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:28:47,900 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd093bcd207e915d77990 HTTP/1.1" 200 None 2026-03-08 01:28:47,901 - lava_callback - DEBUG - Processing node 69acd093bcd207e915d77990: baseline-arm64 2026-03-08 01:28:48,061 - lava_callback - WARNING - Failed to upload log 2026-03-08 01:28:48,131 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:28:48,315 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:28:48,316 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd093bcd207e915d77990/lava-logs.yaml 2026-03-08 01:28:48,320 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:28:48,439 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:28:48,441 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd093bcd207e915d77990/lava_callback.json.gz 2026-03-08 01:28:48,471 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:28:48,497 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd093bcd207e915d77990 HTTP/1.1" 200 None 2026-03-08 01:28:48,499 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:28:48,537 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:28:48,538 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:28:48,560 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd093bcd207e915d77990 HTTP/1.1" 200 None 2026-03-08 01:28:48,632 - lava_callback - INFO - Completed processing callback for node 69acd093bcd207e915d77990 INFO: 172.21.0.1:57182 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38816 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:29:14,143 - lava_callback - INFO - Received callback for node 69aa2dde8c4d487a0e75208b INFO: 172.21.0.1:60672 - "POST /node/69aa2dde8c4d487a0e75208b HTTP/1.0" 202 Accepted 2026-03-08 01:29:14,149 - lava_callback - INFO - Starting async job submit for node 69aa2dde8c4d487a0e75208b 2026-03-08 01:29:14,149 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:29:14,163 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2dde8c4d487a0e75208b HTTP/1.1" 200 None 2026-03-08 01:29:14,164 - lava_callback - DEBUG - Processing node 69aa2dde8c4d487a0e75208b: baseline-arm64-baylibre 2026-03-08 01:29:14,164 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa2dde8c4d487a0e75208b 2026-03-08 01:29:14,165 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:29:14,177 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2dde8c4d487a0e75208b HTTP/1.1" 200 None 2026-03-08 01:29:14,179 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:29:14,192 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a70fafe31b554803095f11 HTTP/1.1" 200 None 2026-03-08 01:29:14,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:29:14,214 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa2dde8c4d487a0e75208b HTTP/1.1" 200 None 2026-03-08 01:29:14,219 - lava_callback - ERROR - Error processing callback for node 69aa2dde8c4d487a0e75208b: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-08 01:29:14,831 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:29:14,848 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:60678 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:60684 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:29:19,970 - lava_callback - INFO - Received callback for node 69acced1bcd207e915d777f4 INFO: 172.21.0.1:38828 - "POST /node/69acced1bcd207e915d777f4 HTTP/1.0" 202 Accepted 2026-03-08 01:29:19,975 - lava_callback - INFO - Starting async job submit for node 69acced1bcd207e915d777f4 2026-03-08 01:29:19,978 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:29:19,993 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acced1bcd207e915d777f4 HTTP/1.1" 200 None 2026-03-08 01:29:19,994 - lava_callback - DEBUG - Processing node 69acced1bcd207e915d777f4: baseline-arm64-foundriesio 2026-03-08 01:29:20,001 - lava_callback - WARNING - Failed to upload log 2026-03-08 01:29:20,007 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:29:20,117 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:29:20,119 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69acced1bcd207e915d777f4/lava-logs.yaml 2026-03-08 01:29:20,120 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:29:20,261 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:29:20,263 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69acced1bcd207e915d777f4/lava_callback.json.gz 2026-03-08 01:29:20,266 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:29:20,280 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acced1bcd207e915d777f4 HTTP/1.1" 200 None 2026-03-08 01:29:20,281 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:29:20,295 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:29:20,296 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:29:20,318 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acced1bcd207e915d777f4 HTTP/1.1" 200 None 2026-03-08 01:29:20,327 - lava_callback - INFO - Completed processing callback for node 69acced1bcd207e915d777f4 INFO: 172.21.0.1:38832 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38838 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38164 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:29:44,849 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:29:44,866 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:44528 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54924 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48328 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:30:09,231 - lava_callback - INFO - Received callback for node 69acd0e4bcd207e915d779b1 INFO: 172.21.0.1:48334 - "POST /node/69acd0e4bcd207e915d779b1 HTTP/1.0" 202 Accepted 2026-03-08 01:30:09,238 - lava_callback - INFO - Starting async job submit for node 69acd0e4bcd207e915d779b1 2026-03-08 01:30:09,267 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:30:09,290 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd0e4bcd207e915d779b1 HTTP/1.1" 200 None 2026-03-08 01:30:09,291 - lava_callback - DEBUG - Processing node 69acd0e4bcd207e915d779b1: baseline-arm64 2026-03-08 01:30:09,525 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:30:09,757 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:30:09,759 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd0e4bcd207e915d779b1/log.txt.gz 2026-03-08 01:30:09,832 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:30:10,008 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:30:10,010 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd0e4bcd207e915d779b1/lava-logs.yaml 2026-03-08 01:30:10,014 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:30:10,160 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:30:10,162 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd0e4bcd207e915d779b1/lava_callback.json.gz 2026-03-08 01:30:10,193 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:30:10,206 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd0e4bcd207e915d779b1 HTTP/1.1" 200 None 2026-03-08 01:30:10,208 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:30:10,223 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:30:10,224 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:30:10,245 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd0e4bcd207e915d779b1 HTTP/1.1" 200 None 2026-03-08 01:30:10,321 - lava_callback - INFO - Completed processing callback for node 69acd0e4bcd207e915d779b1 2026-03-08 01:30:14,868 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:30:14,884 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:56874 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56890 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50096 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50112 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36970 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52632 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48800 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52796 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37700 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:37710 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40186 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40200 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55442 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37494 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42582 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42048 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41526 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:41532 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59700 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59712 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58434 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42410 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:32:56,255 - lava_callback - INFO - Received callback for node 69accdabbcd207e915d77647 INFO: 172.21.0.1:38482 - "POST /node/69accdabbcd207e915d77647 HTTP/1.0" 202 Accepted 2026-03-08 01:32:56,262 - lava_callback - INFO - Starting async job submit for node 69accdabbcd207e915d77647 2026-03-08 01:32:56,315 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:32:56,351 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accdabbcd207e915d77647 HTTP/1.1" 200 None 2026-03-08 01:32:56,352 - lava_callback - DEBUG - Processing node 69accdabbcd207e915d77647: baseline-arm INFO: 172.21.0.1:38490 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:32:56,936 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:32:57,089 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:32:57,091 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69accdabbcd207e915d77647/log.txt.gz 2026-03-08 01:32:57,436 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:32:57,674 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:32:57,677 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69accdabbcd207e915d77647/lava-logs.yaml 2026-03-08 01:32:57,691 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:32:57,910 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:32:57,912 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69accdabbcd207e915d77647/lava_callback.json.gz DEBUG: Setup failed for 69accdabbcd207e915d77647. Transitting job node result: incomplete -> fail 2026-03-08 01:32:58,041 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:32:58,057 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accdabbcd207e915d77647 HTTP/1.1" 200 None 2026-03-08 01:32:58,059 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:32:58,073 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 01:32:58,074 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:32:58,099 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accdabbcd207e915d77647 HTTP/1.1" 200 None 2026-03-08 01:32:58,221 - lava_callback - INFO - Completed processing callback for node 69accdabbcd207e915d77647 INFO: 172.21.0.1:42038 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:33:14,887 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:33:14,910 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:49868 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:49882 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53802 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53808 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56530 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:33:45,559 - lava_callback - INFO - Received callback for node 69accec2bcd207e915d777a1 INFO: 172.21.0.1:40912 - "POST /node/69accec2bcd207e915d777a1 HTTP/1.0" 202 Accepted 2026-03-08 01:33:45,566 - lava_callback - INFO - Starting async job submit for node 69accec2bcd207e915d777a1 2026-03-08 01:33:45,588 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:33:45,610 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accec2bcd207e915d777a1 HTTP/1.1" 200 None 2026-03-08 01:33:45,610 - lava_callback - DEBUG - Processing node 69accec2bcd207e915d777a1: baseline-arm64-broonie 2026-03-08 01:33:45,882 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:33:46,007 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:33:46,009 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69accec2bcd207e915d777a1/log.txt.gz 2026-03-08 01:33:46,165 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:33:46,357 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:33:46,359 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69accec2bcd207e915d777a1/lava-logs.yaml 2026-03-08 01:33:46,365 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:40914 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:33:46,539 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:33:46,541 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69accec2bcd207e915d777a1/lava_callback.json.gz 2026-03-08 01:33:46,564 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:33:46,578 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accec2bcd207e915d777a1 HTTP/1.1" 200 None 2026-03-08 01:33:46,580 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:33:46,593 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:33:46,594 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:33:46,623 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accec2bcd207e915d777a1 HTTP/1.1" 200 None 2026-03-08 01:33:46,625 - lava_callback - INFO - Completed processing callback for node 69accec2bcd207e915d777a1 INFO: 172.21.0.1:42368 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40116 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:34:14,913 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:34:14,928 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:59474 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59486 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52574 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52586 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39970 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56040 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34648 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49034 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:35:13,580 - lava_callback - INFO - Received callback for node 69acd210bcd207e915d77a34 INFO: 172.21.0.1:34814 - "POST /node/69acd210bcd207e915d77a34 HTTP/1.0" 202 Accepted 2026-03-08 01:35:13,586 - lava_callback - INFO - Starting async job submit for node 69acd210bcd207e915d77a34 2026-03-08 01:35:13,607 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:35:13,621 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd210bcd207e915d77a34 HTTP/1.1" 200 None 2026-03-08 01:35:13,621 - lava_callback - DEBUG - Processing node 69acd210bcd207e915d77a34: baseline-arm64-pengutronix 2026-03-08 01:35:13,816 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:35:13,959 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:35:13,961 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69acd210bcd207e915d77a34/log.txt.gz 2026-03-08 01:35:14,089 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:35:14,310 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:35:14,312 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69acd210bcd207e915d77a34/lava-logs.yaml 2026-03-08 01:35:14,318 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:35:14,472 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:35:14,474 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-pengutronix-69acd210bcd207e915d77a34/lava_callback.json.gz 2026-03-08 01:35:14,496 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:35:14,510 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd210bcd207e915d77a34 HTTP/1.1" 200 None 2026-03-08 01:35:14,512 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:35:14,525 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:35:14,527 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:35:14,558 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd210bcd207e915d77a34 HTTP/1.1" 200 None 2026-03-08 01:35:14,560 - lava_callback - INFO - Completed processing callback for node 69acd210bcd207e915d77a34 2026-03-08 01:35:14,930 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:35:14,946 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:34830 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34840 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49454 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49468 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57788 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55866 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51604 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51820 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55286 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55292 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45306 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45310 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35796 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59336 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53168 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54346 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:37:14,710 - lava_callback - INFO - Received callback for node 69aa2fc48c4d487a0e7521b3 INFO: 172.21.0.1:59620 - "POST /node/69aa2fc48c4d487a0e7521b3 HTTP/1.0" 202 Accepted 2026-03-08 01:37:14,718 - lava_callback - INFO - Starting async job submit for node 69aa2fc48c4d487a0e7521b3 2026-03-08 01:37:14,719 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:37:14,744 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2fc48c4d487a0e7521b3 HTTP/1.1" 200 None 2026-03-08 01:37:14,745 - lava_callback - DEBUG - Processing node 69aa2fc48c4d487a0e7521b3: baseline-arm-baylibre 2026-03-08 01:37:14,745 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa2fc48c4d487a0e7521b3 2026-03-08 01:37:14,746 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:37:14,767 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2fc48c4d487a0e7521b3 HTTP/1.1" 200 None 2026-03-08 01:37:14,769 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:37:14,791 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a2340490d011217c9f7d09 HTTP/1.1" 200 None 2026-03-08 01:37:14,792 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:37:14,814 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa2fc48c4d487a0e7521b3 HTTP/1.1" 200 None 2026-03-08 01:37:14,819 - lava_callback - ERROR - Error processing callback for node 69aa2fc48c4d487a0e7521b3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:59624 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59632 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33004 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33012 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47988 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50486 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:37:54,580 - lava_callback - INFO - Received callback for node 69aa2feb8c4d487a0e7521ce INFO: 172.21.0.1:41208 - "POST /node/69aa2feb8c4d487a0e7521ce HTTP/1.0" 202 Accepted 2026-03-08 01:37:54,586 - lava_callback - INFO - Starting async job submit for node 69aa2feb8c4d487a0e7521ce 2026-03-08 01:37:54,586 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:37:54,602 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2feb8c4d487a0e7521ce HTTP/1.1" 200 None 2026-03-08 01:37:54,603 - lava_callback - DEBUG - Processing node 69aa2feb8c4d487a0e7521ce: baseline-arm-baylibre 2026-03-08 01:37:54,603 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa2feb8c4d487a0e7521ce 2026-03-08 01:37:54,603 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:37:54,622 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa2feb8c4d487a0e7521ce HTTP/1.1" 200 None 2026-03-08 01:37:54,623 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:37:54,637 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a2340490d011217c9f7d09 HTTP/1.1" 200 None 2026-03-08 01:37:54,638 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:37:54,658 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa2feb8c4d487a0e7521ce HTTP/1.1" 200 None 2026-03-08 01:37:54,660 - lava_callback - ERROR - Error processing callback for node 69aa2feb8c4d487a0e7521ce: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:41212 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46836 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39312 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:39314 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42692 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42698 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34902 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:38:43,206 - lava_callback - INFO - Received callback for node 69acce99bcd207e915d77716 INFO: 172.21.0.1:36774 - "POST /node/69acce99bcd207e915d77716 HTTP/1.0" 202 Accepted 2026-03-08 01:38:43,212 - lava_callback - INFO - Starting async job submit for node 69acce99bcd207e915d77716 2026-03-08 01:38:43,242 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:38:43,258 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce99bcd207e915d77716 HTTP/1.1" 200 None 2026-03-08 01:38:43,259 - lava_callback - DEBUG - Processing node 69acce99bcd207e915d77716: baseline-arm-clabbe 2026-03-08 01:38:43,534 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:38:43,705 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:38:43,707 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acce99bcd207e915d77716/log.txt.gz 2026-03-08 01:38:43,827 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:38:44,040 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:38:44,042 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acce99bcd207e915d77716/lava-logs.yaml 2026-03-08 01:38:44,047 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:38:44,171 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:38:44,173 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acce99bcd207e915d77716/lava_callback.json.gz DEBUG: Setup failed for 69acce99bcd207e915d77716. Transitting job node result: incomplete -> fail 2026-03-08 01:38:44,237 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:38:44,256 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce99bcd207e915d77716 HTTP/1.1" 200 None 2026-03-08 01:38:44,258 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:38:44,271 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 01:38:44,273 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:38:44,296 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce99bcd207e915d77716 HTTP/1.1" 200 None 2026-03-08 01:38:44,360 - lava_callback - INFO - Completed processing callback for node 69acce99bcd207e915d77716 2026-03-08 01:38:44,949 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:38:44,965 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:36776 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:38:53,885 - lava_callback - INFO - Received callback for node 69accebcbcd207e915d77795 INFO: 172.21.0.1:53454 - "POST /node/69accebcbcd207e915d77795 HTTP/1.0" 202 Accepted 2026-03-08 01:38:53,891 - lava_callback - INFO - Starting async job submit for node 69accebcbcd207e915d77795 2026-03-08 01:38:53,913 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:38:53,927 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accebcbcd207e915d77795 HTTP/1.1" 200 None 2026-03-08 01:38:53,928 - lava_callback - DEBUG - Processing node 69accebcbcd207e915d77795: baseline-arm64-broonie 2026-03-08 01:38:54,189 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:38:54,334 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:38:54,336 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69accebcbcd207e915d77795/log.txt.gz 2026-03-08 01:38:54,489 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:38:54,669 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:38:54,671 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69accebcbcd207e915d77795/lava-logs.yaml 2026-03-08 01:38:54,678 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:38:54,810 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:38:54,812 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69accebcbcd207e915d77795/lava_callback.json.gz 2026-03-08 01:38:54,835 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:38:54,850 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accebcbcd207e915d77795 HTTP/1.1" 200 None 2026-03-08 01:38:54,851 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:38:54,865 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:38:54,867 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:38:54,894 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accebcbcd207e915d77795 HTTP/1.1" 200 None 2026-03-08 01:38:54,895 - lava_callback - INFO - Completed processing callback for node 69accebcbcd207e915d77795 INFO: 172.21.0.1:53462 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58452 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:39:14,792 - lava_callback - INFO - Received callback for node 69aa303a8c4d487a0e7521f7 INFO: 172.21.0.1:53778 - "POST /node/69aa303a8c4d487a0e7521f7 HTTP/1.0" 202 Accepted 2026-03-08 01:39:14,798 - lava_callback - INFO - Starting async job submit for node 69aa303a8c4d487a0e7521f7 2026-03-08 01:39:14,799 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:39:14,813 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa303a8c4d487a0e7521f7 HTTP/1.1" 200 None 2026-03-08 01:39:14,814 - lava_callback - DEBUG - Processing node 69aa303a8c4d487a0e7521f7: baseline-arm-baylibre 2026-03-08 01:39:14,815 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa303a8c4d487a0e7521f7 2026-03-08 01:39:14,815 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:39:14,828 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa303a8c4d487a0e7521f7 HTTP/1.1" 200 None 2026-03-08 01:39:14,829 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:39:14,850 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a2340490d011217c9f7d09 HTTP/1.1" 200 None 2026-03-08 01:39:14,852 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:39:14,872 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa303a8c4d487a0e7521f7 HTTP/1.1" 200 None 2026-03-08 01:39:14,874 - lava_callback - ERROR - Error processing callback for node 69aa303a8c4d487a0e7521f7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-08 01:39:14,967 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:39:15,018 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:53792 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:53794 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53286 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53292 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:39:29,213 - lava_callback - INFO - Received callback for node 69acce9fbcd207e915d77728 INFO: 172.21.0.1:53306 - "POST /node/69acce9fbcd207e915d77728 HTTP/1.0" 202 Accepted 2026-03-08 01:39:29,222 - lava_callback - INFO - Starting async job submit for node 69acce9fbcd207e915d77728 2026-03-08 01:39:29,266 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:39:29,285 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce9fbcd207e915d77728 HTTP/1.1" 200 None 2026-03-08 01:39:29,286 - lava_callback - DEBUG - Processing node 69acce9fbcd207e915d77728: baseline-nfs-arm64 2026-03-08 01:39:30,416 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:39:30,623 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:39:30,626 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acce9fbcd207e915d77728/log.txt.gz 2026-03-08 01:39:31,257 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:39:31,520 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:39:31,523 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acce9fbcd207e915d77728/lava-logs.yaml 2026-03-08 01:39:31,545 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:39:31,732 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:39:31,734 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acce9fbcd207e915d77728/lava_callback.json.gz DEBUG: Setup failed for 69acce9fbcd207e915d77728. Transitting job node result: incomplete -> fail 2026-03-08 01:39:31,821 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:39:31,839 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acce9fbcd207e915d77728 HTTP/1.1" 200 None 2026-03-08 01:39:31,841 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:39:31,858 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:39:31,860 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:39:31,887 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acce9fbcd207e915d77728 HTTP/1.1" 200 None 2026-03-08 01:39:32,007 - lava_callback - INFO - Completed processing callback for node 69acce9fbcd207e915d77728 INFO: 172.21.0.1:48136 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:39:45,021 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:39:45,047 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:47702 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59854 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60006 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:40:14,704 - lava_callback - INFO - Received callback for node 69aa306a8c4d487a0e752214 INFO: 172.21.0.1:59102 - "POST /node/69aa306a8c4d487a0e752214 HTTP/1.0" 202 Accepted 2026-03-08 01:40:14,709 - lava_callback - INFO - Starting async job submit for node 69aa306a8c4d487a0e752214 2026-03-08 01:40:14,710 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:40:14,725 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa306a8c4d487a0e752214 HTTP/1.1" 200 None 2026-03-08 01:40:14,726 - lava_callback - DEBUG - Processing node 69aa306a8c4d487a0e752214: baseline-arm64-baylibre 2026-03-08 01:40:14,726 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa306a8c4d487a0e752214 2026-03-08 01:40:14,726 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:40:14,753 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa306a8c4d487a0e752214 HTTP/1.1" 200 None 2026-03-08 01:40:14,755 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:40:14,768 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6d3423d518458fb340278 HTTP/1.1" 200 None 2026-03-08 01:40:14,770 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:40:14,790 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa306a8c4d487a0e752214 HTTP/1.1" 200 None 2026-03-08 01:40:14,795 - lava_callback - ERROR - Error processing callback for node 69aa306a8c4d487a0e752214: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:59106 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59118 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44198 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44204 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55556 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46274 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44682 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58860 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36210 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:36218 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:41:19,496 - lava_callback - INFO - Received callback for node 69acd39dbcd207e915d77af2 INFO: 172.21.0.1:36230 - "POST /node/69acd39dbcd207e915d77af2 HTTP/1.0" 202 Accepted 2026-03-08 01:41:19,502 - lava_callback - INFO - Starting async job submit for node 69acd39dbcd207e915d77af2 2026-03-08 01:41:19,504 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:41:19,518 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd39dbcd207e915d77af2 HTTP/1.1" 200 None 2026-03-08 01:41:19,519 - lava_callback - DEBUG - Processing node 69acd39dbcd207e915d77af2: baseline-nfs-arm64 2026-03-08 01:41:19,520 - lava_callback - WARNING - Failed to upload log 2026-03-08 01:41:19,521 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:41:19,659 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:41:19,661 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acd39dbcd207e915d77af2/lava-logs.yaml 2026-03-08 01:41:19,662 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:41:19,827 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:41:19,828 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acd39dbcd207e915d77af2/lava_callback.json.gz 2026-03-08 01:41:19,830 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:41:19,844 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd39dbcd207e915d77af2 HTTP/1.1" 200 None 2026-03-08 01:41:19,846 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:41:19,859 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:41:19,861 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:41:19,881 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd39dbcd207e915d77af2 HTTP/1.1" 200 None 2026-03-08 01:41:19,888 - lava_callback - INFO - Completed processing callback for node 69acd39dbcd207e915d77af2 INFO: 172.21.0.1:35020 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35034 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42766 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:41:45,049 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:41:45,071 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:44424 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55488 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59712 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47298 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47300 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60024 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60036 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35438 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48826 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:42:54,918 - lava_callback - INFO - Received callback for node 69aa310b8c4d487a0e752257 INFO: 172.21.0.1:45140 - "POST /node/69aa310b8c4d487a0e752257 HTTP/1.0" 202 Accepted 2026-03-08 01:42:54,923 - lava_callback - INFO - Starting async job submit for node 69aa310b8c4d487a0e752257 2026-03-08 01:42:54,924 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:42:54,939 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa310b8c4d487a0e752257 HTTP/1.1" 200 None 2026-03-08 01:42:54,940 - lava_callback - DEBUG - Processing node 69aa310b8c4d487a0e752257: baseline-arm64-baylibre 2026-03-08 01:42:54,940 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa310b8c4d487a0e752257 2026-03-08 01:42:54,940 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:42:54,954 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa310b8c4d487a0e752257 HTTP/1.1" 200 None 2026-03-08 01:42:54,956 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:42:54,970 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6d3423d518458fb340278 HTTP/1.1" 200 None 2026-03-08 01:42:54,972 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:42:54,997 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa310b8c4d487a0e752257 HTTP/1.1" 200 None 2026-03-08 01:42:54,999 - lava_callback - ERROR - Error processing callback for node 69aa310b8c4d487a0e752257: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:45142 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37904 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37974 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:37988 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45154 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45158 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42616 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59498 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35544 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37466 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38730 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:38732 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55580 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55594 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35436 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35448 - "GET /node/69939933cebe99dd8e074de0%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.21.0.1:38796 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41328 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47268 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:45:15,075 - lava_callback - INFO - Received callback for node 69aa31a28c4d487a0e75228f INFO: 172.21.0.1:59200 - "POST /node/69aa31a28c4d487a0e75228f HTTP/1.0" 202 Accepted 2026-03-08 01:45:15,080 - lava_callback - INFO - Starting async job submit for node 69aa31a28c4d487a0e75228f 2026-03-08 01:45:15,081 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:45:15,096 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa31a28c4d487a0e75228f HTTP/1.1" 200 None 2026-03-08 01:45:15,097 - lava_callback - DEBUG - Processing node 69aa31a28c4d487a0e75228f: baseline-arm-baylibre 2026-03-08 01:45:15,097 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa31a28c4d487a0e75228f 2026-03-08 01:45:15,098 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:45:15,123 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa31a28c4d487a0e75228f HTTP/1.1" 200 None 2026-03-08 01:45:15,125 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:45:15,138 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a2340490d011217c9f7d09 HTTP/1.1" 200 None 2026-03-08 01:45:15,140 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:45:15,160 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa31a28c4d487a0e75228f HTTP/1.1" 200 None 2026-03-08 01:45:15,166 - lava_callback - ERROR - Error processing callback for node 69aa31a28c4d487a0e75228f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:59212 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59222 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46068 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46074 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55572 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57970 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48060 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42816 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:46:15,052 - lava_callback - INFO - Received callback for node 69aa31d28c4d487a0e7522a0 INFO: 172.21.0.1:52530 - "POST /node/69aa31d28c4d487a0e7522a0 HTTP/1.0" 202 Accepted 2026-03-08 01:46:15,059 - lava_callback - INFO - Starting async job submit for node 69aa31d28c4d487a0e7522a0 2026-03-08 01:46:15,060 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:46:15,074 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa31d28c4d487a0e7522a0 HTTP/1.1" 200 None 2026-03-08 01:46:15,075 - lava_callback - DEBUG - Processing node 69aa31d28c4d487a0e7522a0: baseline-arm64-baylibre 2026-03-08 01:46:15,075 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa31d28c4d487a0e7522a0 2026-03-08 01:46:15,076 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:46:15,103 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa31d28c4d487a0e7522a0 HTTP/1.1" 200 None 2026-03-08 01:46:15,104 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:46:15,117 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6d3423d518458fb340278 HTTP/1.1" 200 None 2026-03-08 01:46:15,118 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:46:15,138 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa31d28c4d487a0e7522a0 HTTP/1.1" 200 None 2026-03-08 01:46:15,140 - lava_callback - ERROR - Error processing callback for node 69aa31d28c4d487a0e7522a0: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:52546 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52558 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51504 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:46:25,788 - lava_callback - INFO - Received callback for node 69acd4c8bcd207e915d77b66 INFO: 172.21.0.1:51514 - "POST /node/69acd4c8bcd207e915d77b66 HTTP/1.0" 202 Accepted 2026-03-08 01:46:25,795 - lava_callback - INFO - Starting async job submit for node 69acd4c8bcd207e915d77b66 2026-03-08 01:46:25,826 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:46:25,845 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd4c8bcd207e915d77b66 HTTP/1.1" 200 None 2026-03-08 01:46:25,845 - lava_callback - DEBUG - Processing node 69acd4c8bcd207e915d77b66: baseline-arm64 2026-03-08 01:46:26,002 - lava_callback - WARNING - Failed to upload log 2026-03-08 01:46:26,071 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:46:26,297 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:46:26,299 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd4c8bcd207e915d77b66/lava-logs.yaml 2026-03-08 01:46:26,304 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:46:26,510 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:46:26,512 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd4c8bcd207e915d77b66/lava_callback.json.gz INFO: 172.21.0.1:51522 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:46:26,560 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:46:26,613 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd4c8bcd207e915d77b66 HTTP/1.1" 200 None 2026-03-08 01:46:26,615 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:46:26,639 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:46:26,640 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:46:26,661 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd4c8bcd207e915d77b66 HTTP/1.1" 200 None 2026-03-08 01:46:26,742 - lava_callback - INFO - Completed processing callback for node 69acd4c8bcd207e915d77b66 INFO: 172.21.0.1:49608 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:46:45,073 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:46:45,090 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:40198 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42672 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50524 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:47:15,133 - lava_callback - INFO - Received callback for node 69aa32198c4d487a0e7522c1 INFO: 172.21.0.1:54666 - "POST /node/69aa32198c4d487a0e7522c1 HTTP/1.0" 202 Accepted 2026-03-08 01:47:15,139 - lava_callback - INFO - Starting async job submit for node 69aa32198c4d487a0e7522c1 2026-03-08 01:47:15,140 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:47:15,155 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa32198c4d487a0e7522c1 HTTP/1.1" 200 None 2026-03-08 01:47:15,156 - lava_callback - DEBUG - Processing node 69aa32198c4d487a0e7522c1: baseline-arm-baylibre 2026-03-08 01:47:15,156 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa32198c4d487a0e7522c1 2026-03-08 01:47:15,157 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.21.0.1:54680 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 01:47:15,170 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa32198c4d487a0e7522c1 HTTP/1.1" 200 None 2026-03-08 01:47:15,172 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:47:15,185 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a2340490d011217c9f7d09 HTTP/1.1" 200 None 2026-03-08 01:47:15,187 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:47:15,215 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa32198c4d487a0e7522c1 HTTP/1.1" 200 None 2026-03-08 01:47:15,217 - lava_callback - ERROR - Error processing callback for node 69aa32198c4d487a0e7522c1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:54696 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:47:18,124 - lava_callback - INFO - Received callback for node 69acd4c4bcd207e915d77b5f INFO: 172.21.0.1:54700 - "POST /node/69acd4c4bcd207e915d77b5f HTTP/1.0" 202 Accepted 2026-03-08 01:47:18,130 - lava_callback - INFO - Starting async job submit for node 69acd4c4bcd207e915d77b5f 2026-03-08 01:47:18,148 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:47:18,162 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd4c4bcd207e915d77b5f HTTP/1.1" 200 None 2026-03-08 01:47:18,163 - lava_callback - DEBUG - Processing node 69acd4c4bcd207e915d77b5f: baseline-x86 2026-03-08 01:47:18,423 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:47:18,550 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:47:18,552 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-x86-69acd4c4bcd207e915d77b5f/log.txt.gz 2026-03-08 01:47:18,708 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:47:18,912 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:47:18,914 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-x86-69acd4c4bcd207e915d77b5f/lava-logs.yaml 2026-03-08 01:47:18,921 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:47:19,083 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:47:19,085 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-x86-69acd4c4bcd207e915d77b5f/lava_callback.json.gz 2026-03-08 01:47:19,104 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:47:19,118 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd4c4bcd207e915d77b5f HTTP/1.1" 200 None 2026-03-08 01:47:19,120 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:47:19,138 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc037bcd207e915d76bc8 HTTP/1.1" 200 None 2026-03-08 01:47:19,140 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:47:19,163 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd4c4bcd207e915d77b5f HTTP/1.1" 200 None 2026-03-08 01:47:19,169 - lava_callback - INFO - Completed processing callback for node 69acd4c4bcd207e915d77b5f INFO: 172.21.0.1:49888 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49904 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47968 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:47:45,092 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:47:45,109 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:53378 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:47:55,165 - lava_callback - INFO - Received callback for node 69aa32408c4d487a0e7522d2 INFO: 172.21.0.1:47722 - "POST /node/69aa32408c4d487a0e7522d2 HTTP/1.0" 202 Accepted 2026-03-08 01:47:55,173 - lava_callback - INFO - Starting async job submit for node 69aa32408c4d487a0e7522d2 2026-03-08 01:47:55,174 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:47:55,190 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa32408c4d487a0e7522d2 HTTP/1.1" 200 None 2026-03-08 01:47:55,191 - lava_callback - DEBUG - Processing node 69aa32408c4d487a0e7522d2: baseline-arm-baylibre 2026-03-08 01:47:55,191 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa32408c4d487a0e7522d2 2026-03-08 01:47:55,192 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:47:55,209 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa32408c4d487a0e7522d2 HTTP/1.1" 200 None 2026-03-08 01:47:55,211 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:47:55,223 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a2340490d011217c9f7d09 HTTP/1.1" 200 None 2026-03-08 01:47:55,225 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:47:55,244 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa32408c4d487a0e7522d2 HTTP/1.1" 200 None 2026-03-08 01:47:55,251 - lava_callback - ERROR - Error processing callback for node 69aa32408c4d487a0e7522d2: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:47738 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34350 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43098 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:43104 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38704 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38708 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52182 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39412 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58604 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38366 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35138 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:35142 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56124 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:49:26,498 - lava_callback - INFO - Received callback for node 69acd3ebbcd207e915d77b05 INFO: 172.21.0.1:56138 - "POST /node/69acd3ebbcd207e915d77b05 HTTP/1.0" 202 Accepted 2026-03-08 01:49:26,505 - lava_callback - INFO - Starting async job submit for node 69acd3ebbcd207e915d77b05 2026-03-08 01:49:26,540 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.21.0.1:56144 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:49:26,554 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd3ebbcd207e915d77b05 HTTP/1.1" 200 None 2026-03-08 01:49:26,555 - lava_callback - DEBUG - Processing node 69acd3ebbcd207e915d77b05: baseline-arm64 2026-03-08 01:49:26,949 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:49:27,247 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:49:27,249 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd3ebbcd207e915d77b05/log.txt.gz 2026-03-08 01:49:27,463 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:49:27,659 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:49:27,661 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd3ebbcd207e915d77b05/lava-logs.yaml 2026-03-08 01:49:27,670 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:49:27,827 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:49:27,829 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd3ebbcd207e915d77b05/lava_callback.json.gz DEBUG: Setup failed for 69acd3ebbcd207e915d77b05. Transitting job node result: incomplete -> fail 2026-03-08 01:49:27,915 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:49:27,930 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd3ebbcd207e915d77b05 HTTP/1.1" 200 None 2026-03-08 01:49:27,931 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:49:27,945 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:49:27,946 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:49:27,969 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd3ebbcd207e915d77b05 HTTP/1.1" 200 None 2026-03-08 01:49:28,045 - lava_callback - INFO - Completed processing callback for node 69acd3ebbcd207e915d77b05 INFO: 172.21.0.1:39164 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:49:45,112 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:49:45,136 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:59336 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:49:55,312 - lava_callback - INFO - Received callback for node 69aa32c18c4d487a0e752309 INFO: 172.21.0.1:37232 - "POST /node/69aa32c18c4d487a0e752309 HTTP/1.0" 202 Accepted 2026-03-08 01:49:55,318 - lava_callback - INFO - Starting async job submit for node 69aa32c18c4d487a0e752309 2026-03-08 01:49:55,319 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:49:55,335 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa32c18c4d487a0e752309 HTTP/1.1" 200 None 2026-03-08 01:49:55,336 - lava_callback - DEBUG - Processing node 69aa32c18c4d487a0e752309: baseline-arm64-baylibre 2026-03-08 01:49:55,336 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa32c18c4d487a0e752309 2026-03-08 01:49:55,337 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:49:55,351 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa32c18c4d487a0e752309 HTTP/1.1" 200 None 2026-03-08 01:49:55,353 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:49:55,374 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6d3423d518458fb340278 HTTP/1.1" 200 None 2026-03-08 01:49:55,376 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:49:55,405 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa32c18c4d487a0e752309 HTTP/1.1" 200 None 2026-03-08 01:49:55,407 - lava_callback - ERROR - Error processing callback for node 69aa32c18c4d487a0e752309: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:37248 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57662 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51220 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 01:50:15,707 - lava_callback - INFO - Received callback for node 69acd13cbcd207e915d779d6 INFO: 172.21.0.1:51228 - "POST /node/69acd13cbcd207e915d779d6 HTTP/1.0" 202 Accepted 2026-03-08 01:50:15,757 - lava_callback - INFO - Starting async job submit for node 69acd13cbcd207e915d779d6 2026-03-08 01:50:15,798 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:50:15,831 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd13cbcd207e915d779d6 HTTP/1.1" 200 None 2026-03-08 01:50:15,833 - lava_callback - DEBUG - Processing node 69acd13cbcd207e915d779d6: baseline-arm-pengutronix 2026-03-08 01:50:16,017 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:50:16,151 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:50:16,153 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69acd13cbcd207e915d779d6/log.txt.gz 2026-03-08 01:50:16,220 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:51244 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:50:16,612 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:50:16,614 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69acd13cbcd207e915d779d6/lava-logs.yaml 2026-03-08 01:50:16,618 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:50:16,787 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:50:16,794 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69acd13cbcd207e915d779d6/lava_callback.json.gz DEBUG: Setup failed for 69acd13cbcd207e915d779d6. Transitting job node result: incomplete -> fail 2026-03-08 01:50:16,844 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:50:16,860 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd13cbcd207e915d779d6 HTTP/1.1" 200 None 2026-03-08 01:50:16,861 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:50:16,878 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 01:50:16,880 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:50:16,922 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd13cbcd207e915d779d6 HTTP/1.1" 200 None 2026-03-08 01:50:17,029 - lava_callback - INFO - Completed processing callback for node 69acd13cbcd207e915d779d6 INFO: 172.21.0.1:57824 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57828 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59404 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:50:45,138 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:50:45,155 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:40812 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:50:48,617 - lava_callback - INFO - Received callback for node 69acd5bdbcd207e915d77bd2 INFO: 172.21.0.1:40816 - "POST /node/69acd5bdbcd207e915d77bd2 HTTP/1.0" 202 Accepted 2026-03-08 01:50:48,629 - lava_callback - INFO - Starting async job submit for node 69acd5bdbcd207e915d77bd2 2026-03-08 01:50:48,680 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:50:48,705 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd5bdbcd207e915d77bd2 HTTP/1.1" 200 None 2026-03-08 01:50:48,706 - lava_callback - DEBUG - Processing node 69acd5bdbcd207e915d77bd2: baseline-arm64 2026-03-08 01:50:48,868 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:50:49,419 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:50:49,421 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd5bdbcd207e915d77bd2/log.txt.gz 2026-03-08 01:50:49,545 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:50:50,053 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:50:50,060 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd5bdbcd207e915d77bd2/lava-logs.yaml 2026-03-08 01:50:50,066 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:50:50,246 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:50:50,248 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd5bdbcd207e915d77bd2/lava_callback.json.gz 2026-03-08 01:50:50,280 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:50:50,304 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd5bdbcd207e915d77bd2 HTTP/1.1" 200 None 2026-03-08 01:50:50,306 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:50:50,320 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:50:50,321 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:50:50,345 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd5bdbcd207e915d77bd2 HTTP/1.1" 200 None 2026-03-08 01:50:50,433 - lava_callback - INFO - Completed processing callback for node 69acd5bdbcd207e915d77bd2 INFO: 172.21.0.1:48466 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40154 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:51:15,157 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.21.0.1:53200 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 01:51:15,180 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:53208 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59588 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59592 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35188 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35830 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39154 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34818 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58584 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58592 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39730 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:52:24,842 - lava_callback - INFO - Received callback for node 69accec1bcd207e915d7779e INFO: 172.21.0.1:39732 - "POST /node/69accec1bcd207e915d7779e HTTP/1.0" 202 Accepted 2026-03-08 01:52:24,850 - lava_callback - INFO - Starting async job submit for node 69accec1bcd207e915d7779e 2026-03-08 01:52:24,873 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:52:24,887 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accec1bcd207e915d7779e HTTP/1.1" 200 None 2026-03-08 01:52:24,888 - lava_callback - DEBUG - Processing node 69accec1bcd207e915d7779e: baseline-arm64-broonie 2026-03-08 01:52:25,147 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:52:25,267 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:52:25,269 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69accec1bcd207e915d7779e/log.txt.gz 2026-03-08 01:52:25,420 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:52:25,587 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:52:25,590 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69accec1bcd207e915d7779e/lava-logs.yaml 2026-03-08 01:52:25,596 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:52:25,747 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:52:25,750 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69accec1bcd207e915d7779e/lava_callback.json.gz 2026-03-08 01:52:25,778 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:52:25,803 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accec1bcd207e915d7779e HTTP/1.1" 200 None 2026-03-08 01:52:25,805 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:52:25,826 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:52:25,828 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:52:25,861 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accec1bcd207e915d7779e HTTP/1.1" 200 None 2026-03-08 01:52:25,863 - lava_callback - INFO - Completed processing callback for node 69accec1bcd207e915d7779e INFO: 172.21.0.1:39738 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:52:35,490 - lava_callback - INFO - Received callback for node 69aa33588c4d487a0e75233f INFO: 172.21.0.1:51728 - "POST /node/69aa33588c4d487a0e75233f HTTP/1.0" 202 Accepted 2026-03-08 01:52:35,496 - lava_callback - INFO - Starting async job submit for node 69aa33588c4d487a0e75233f 2026-03-08 01:52:35,497 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:52:35,515 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa33588c4d487a0e75233f HTTP/1.1" 200 None 2026-03-08 01:52:35,516 - lava_callback - DEBUG - Processing node 69aa33588c4d487a0e75233f: baseline-arm64-baylibre 2026-03-08 01:52:35,516 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa33588c4d487a0e75233f 2026-03-08 01:52:35,517 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:52:35,530 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa33588c4d487a0e75233f HTTP/1.1" 200 None 2026-03-08 01:52:35,531 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:52:35,546 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6d3423d518458fb340278 HTTP/1.1" 200 None 2026-03-08 01:52:35,548 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:52:35,575 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa33588c4d487a0e75233f HTTP/1.1" 200 None 2026-03-08 01:52:35,576 - lava_callback - ERROR - Error processing callback for node 69aa33588c4d487a0e75233f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:51734 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:52:45,182 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:52:45,198 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:52422 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38444 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42882 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53088 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:53100 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55304 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55318 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41596 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34016 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58710 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39380 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33646 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:33650 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46820 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46832 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49464 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47772 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44164 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49286 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41448 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 01:55:15,658 - lava_callback - INFO - Received callback for node 69aa34018c4d487a0e75237d INFO: 172.21.0.1:41456 - "POST /node/69aa34018c4d487a0e75237d HTTP/1.0" 202 Accepted 2026-03-08 01:55:15,664 - lava_callback - INFO - Starting async job submit for node 69aa34018c4d487a0e75237d 2026-03-08 01:55:15,665 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:55:15,679 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa34018c4d487a0e75237d HTTP/1.1" 200 None 2026-03-08 01:55:15,680 - lava_callback - DEBUG - Processing node 69aa34018c4d487a0e75237d: baseline-arm64-baylibre 2026-03-08 01:55:15,680 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa34018c4d487a0e75237d 2026-03-08 01:55:15,680 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:55:15,726 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa34018c4d487a0e75237d HTTP/1.1" 200 None 2026-03-08 01:55:15,728 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:55:15,753 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6d3423d518458fb340278 HTTP/1.1" 200 None 2026-03-08 01:55:15,755 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:55:15,775 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa34018c4d487a0e75237d HTTP/1.1" 200 None 2026-03-08 01:55:15,776 - lava_callback - ERROR - Error processing callback for node 69aa34018c4d487a0e75237d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:41464 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36252 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36254 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46062 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58222 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:55:47,234 - lava_callback - INFO - Received callback for node 69acd705bcd207e915d77c56 INFO: 172.21.0.1:58230 - "POST /node/69acd705bcd207e915d77c56 HTTP/1.0" 202 Accepted 2026-03-08 01:55:47,239 - lava_callback - INFO - Starting async job submit for node 69acd705bcd207e915d77c56 2026-03-08 01:55:47,241 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:55:47,256 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd705bcd207e915d77c56 HTTP/1.1" 200 None 2026-03-08 01:55:47,257 - lava_callback - DEBUG - Processing node 69acd705bcd207e915d77c56: baseline-nfs-arm64 2026-03-08 01:55:47,258 - lava_callback - WARNING - Failed to upload log 2026-03-08 01:55:47,259 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:55:47,458 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:55:47,460 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acd705bcd207e915d77c56/lava-logs.yaml 2026-03-08 01:55:47,461 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:55:47,610 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:55:47,611 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acd705bcd207e915d77c56/lava_callback.json.gz 2026-03-08 01:55:47,613 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:55:47,628 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd705bcd207e915d77c56 HTTP/1.1" 200 None 2026-03-08 01:55:47,629 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:55:47,644 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:55:47,645 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:55:47,671 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd705bcd207e915d77c56 HTTP/1.1" 200 None 2026-03-08 01:55:47,678 - lava_callback - INFO - Completed processing callback for node 69acd705bcd207e915d77c56 INFO: 172.21.0.1:44882 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54666 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34078 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 01:56:15,201 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:56:15,229 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:34086 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54384 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54388 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:56:35,687 - lava_callback - INFO - Received callback for node 69aa34468c4d487a0e7523aa INFO: 172.21.0.1:36628 - "POST /node/69aa34468c4d487a0e7523aa HTTP/1.0" 202 Accepted 2026-03-08 01:56:35,692 - lava_callback - INFO - Starting async job submit for node 69aa34468c4d487a0e7523aa 2026-03-08 01:56:35,693 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:56:35,718 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa34468c4d487a0e7523aa HTTP/1.1" 200 None 2026-03-08 01:56:35,719 - lava_callback - DEBUG - Processing node 69aa34468c4d487a0e7523aa: baseline-arm-baylibre 2026-03-08 01:56:35,719 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa34468c4d487a0e7523aa 2026-03-08 01:56:35,720 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:56:35,734 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa34468c4d487a0e7523aa HTTP/1.1" 200 None 2026-03-08 01:56:35,736 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:56:35,752 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a4d6d27caa0b8e1b59d8ae HTTP/1.1" 200 None 2026-03-08 01:56:35,754 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:56:35,778 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa34468c4d487a0e7523aa HTTP/1.1" 200 None 2026-03-08 01:56:35,779 - lava_callback - ERROR - Error processing callback for node 69aa34468c4d487a0e7523aa: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:36634 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51208 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43252 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:56:57,182 - lava_callback - INFO - Received callback for node 69acd341bcd207e915d77ac1 INFO: 172.21.0.1:43268 - "POST /node/69acd341bcd207e915d77ac1 HTTP/1.0" 202 Accepted 2026-03-08 01:56:57,188 - lava_callback - INFO - Starting async job submit for node 69acd341bcd207e915d77ac1 2026-03-08 01:56:57,246 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:56:57,265 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd341bcd207e915d77ac1 HTTP/1.1" 200 None 2026-03-08 01:56:57,266 - lava_callback - DEBUG - Processing node 69acd341bcd207e915d77ac1: ltp-smoke 2026-03-08 01:56:57,759 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:56:57,934 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:56:57,936 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69acd341bcd207e915d77ac1/log.txt.gz 2026-03-08 01:56:58,208 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:56:58,427 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:56:58,430 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69acd341bcd207e915d77ac1/lava-logs.yaml 2026-03-08 01:56:58,441 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:56:58,622 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:56:58,624 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69acd341bcd207e915d77ac1/lava_callback.json.gz 2026-03-08 01:56:58,742 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:56:58,757 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd341bcd207e915d77ac1 HTTP/1.1" 200 None 2026-03-08 01:56:58,759 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:56:58,772 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc012bcd207e915d76b9c HTTP/1.1" 200 None 2026-03-08 01:56:58,773 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:56:58,802 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd341bcd207e915d77ac1 HTTP/1.1" 200 None 2026-03-08 01:56:58,933 - lava_callback - INFO - Completed processing callback for node 69acd341bcd207e915d77ac1 INFO: 172.21.0.1:45872 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58650 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 01:57:15,231 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:57:15,247 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-08 01:57:15,608 - lava_callback - INFO - Received callback for node 69aa346d8c4d487a0e7523b9 INFO: 172.21.0.1:58664 - "POST /node/69aa346d8c4d487a0e7523b9 HTTP/1.0" 202 Accepted 2026-03-08 01:57:15,614 - lava_callback - INFO - Starting async job submit for node 69aa346d8c4d487a0e7523b9 2026-03-08 01:57:15,615 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:57:15,629 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa346d8c4d487a0e7523b9 HTTP/1.1" 200 None 2026-03-08 01:57:15,630 - lava_callback - DEBUG - Processing node 69aa346d8c4d487a0e7523b9: baseline-arm-baylibre 2026-03-08 01:57:15,630 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa346d8c4d487a0e7523b9 2026-03-08 01:57:15,631 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:57:15,643 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa346d8c4d487a0e7523b9 HTTP/1.1" 200 None 2026-03-08 01:57:15,645 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:57:15,658 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a4d6d27caa0b8e1b59d8ae HTTP/1.1" 200 None 2026-03-08 01:57:15,660 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:57:15,680 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa346d8c4d487a0e7523b9 HTTP/1.1" 200 None 2026-03-08 01:57:15,682 - lava_callback - ERROR - Error processing callback for node 69aa346d8c4d487a0e7523b9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:58670 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60106 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60110 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:57:28,593 - lava_callback - INFO - Received callback for node 69acd74dbcd207e915d77c68 INFO: 172.21.0.1:60116 - "POST /node/69acd74dbcd207e915d77c68 HTTP/1.0" 202 Accepted 2026-03-08 01:57:28,600 - lava_callback - INFO - Starting async job submit for node 69acd74dbcd207e915d77c68 2026-03-08 01:57:28,629 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:57:28,643 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd74dbcd207e915d77c68 HTTP/1.1" 200 None 2026-03-08 01:57:28,644 - lava_callback - DEBUG - Processing node 69acd74dbcd207e915d77c68: baseline-arm64 2026-03-08 01:57:28,775 - lava_callback - WARNING - Failed to upload log 2026-03-08 01:57:28,843 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:57:29,013 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:57:29,016 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd74dbcd207e915d77c68/lava-logs.yaml 2026-03-08 01:57:29,020 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:57:29,191 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:57:29,193 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd74dbcd207e915d77c68/lava_callback.json.gz 2026-03-08 01:57:29,244 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:57:29,268 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd74dbcd207e915d77c68 HTTP/1.1" 200 None 2026-03-08 01:57:29,270 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:57:29,291 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:57:29,292 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:57:29,322 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd74dbcd207e915d77c68 HTTP/1.1" 200 None 2026-03-08 01:57:29,392 - lava_callback - INFO - Completed processing callback for node 69acd74dbcd207e915d77c68 INFO: 172.21.0.1:42150 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:57:45,249 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:57:45,267 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 2026-03-08 01:57:46,155 - lava_callback - INFO - Received callback for node 69acd584bcd207e915d77bb8 INFO: 172.21.0.1:37234 - "POST /node/69acd584bcd207e915d77bb8 HTTP/1.0" 202 Accepted 2026-03-08 01:57:46,161 - lava_callback - INFO - Starting async job submit for node 69acd584bcd207e915d77bb8 2026-03-08 01:57:46,164 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:57:46,179 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd584bcd207e915d77bb8 HTTP/1.1" 200 None 2026-03-08 01:57:46,180 - lava_callback - DEBUG - Processing node 69acd584bcd207e915d77bb8: baseline-arm64-foundriesio 2026-03-08 01:57:46,187 - lava_callback - WARNING - Failed to upload log 2026-03-08 01:57:46,192 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:57:46,328 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:57:46,330 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69acd584bcd207e915d77bb8/lava-logs.yaml 2026-03-08 01:57:46,331 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:57:46,462 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:57:46,464 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69acd584bcd207e915d77bb8/lava_callback.json.gz 2026-03-08 01:57:46,467 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:57:46,482 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd584bcd207e915d77bb8 HTTP/1.1" 200 None 2026-03-08 01:57:46,483 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:57:46,501 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:57:46,502 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.21.0.1:37236 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:57:46,524 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd584bcd207e915d77bb8 HTTP/1.1" 200 None 2026-03-08 01:57:46,534 - lava_callback - INFO - Completed processing callback for node 69acd584bcd207e915d77bb8 2026-03-08 01:57:55,752 - lava_callback - INFO - Received callback for node 69aa34958c4d487a0e7523cf INFO: 172.21.0.1:51206 - "POST /node/69aa34958c4d487a0e7523cf HTTP/1.0" 202 Accepted 2026-03-08 01:57:55,758 - lava_callback - INFO - Starting async job submit for node 69aa34958c4d487a0e7523cf 2026-03-08 01:57:55,759 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:57:55,777 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa34958c4d487a0e7523cf HTTP/1.1" 200 None 2026-03-08 01:57:55,777 - lava_callback - DEBUG - Processing node 69aa34958c4d487a0e7523cf: baseline-arm-baylibre 2026-03-08 01:57:55,778 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa34958c4d487a0e7523cf 2026-03-08 01:57:55,778 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:57:55,792 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa34958c4d487a0e7523cf HTTP/1.1" 200 None 2026-03-08 01:57:55,793 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:57:55,806 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a4d6d27caa0b8e1b59d8ae HTTP/1.1" 200 None 2026-03-08 01:57:55,807 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:57:55,827 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa34958c4d487a0e7523cf HTTP/1.1" 200 None 2026-03-08 01:57:55,832 - lava_callback - ERROR - Error processing callback for node 69aa34958c4d487a0e7523cf: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:51208 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52468 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39908 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 01:58:15,269 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:58:15,285 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:39912 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41142 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41156 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:58:35,711 - lava_callback - INFO - Received callback for node 69aa34bb8c4d487a0e7523e4 INFO: 172.21.0.1:49146 - "POST /node/69aa34bb8c4d487a0e7523e4 HTTP/1.0" 202 Accepted 2026-03-08 01:58:35,717 - lava_callback - INFO - Starting async job submit for node 69aa34bb8c4d487a0e7523e4 2026-03-08 01:58:35,718 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:58:35,733 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa34bb8c4d487a0e7523e4 HTTP/1.1" 200 None 2026-03-08 01:58:35,734 - lava_callback - DEBUG - Processing node 69aa34bb8c4d487a0e7523e4: baseline-arm-baylibre 2026-03-08 01:58:35,734 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa34bb8c4d487a0e7523e4 2026-03-08 01:58:35,735 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:58:35,754 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa34bb8c4d487a0e7523e4 HTTP/1.1" 200 None 2026-03-08 01:58:35,756 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:58:35,792 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a4d6d27caa0b8e1b59d8ae HTTP/1.1" 200 None 2026-03-08 01:58:35,794 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:58:35,815 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa34bb8c4d487a0e7523e4 HTTP/1.1" 200 None 2026-03-08 01:58:35,820 - lava_callback - ERROR - Error processing callback for node 69aa34bb8c4d487a0e7523e4: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:49152 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32970 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40242 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39356 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35860 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 01:59:15,756 - lava_callback - INFO - Received callback for node 69aa34e38c4d487a0e7523f1 INFO: 172.21.0.1:35870 - "POST /node/69aa34e38c4d487a0e7523f1 HTTP/1.0" 202 Accepted 2026-03-08 01:59:15,762 - lava_callback - INFO - Starting async job submit for node 69aa34e38c4d487a0e7523f1 2026-03-08 01:59:15,763 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:59:15,776 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa34e38c4d487a0e7523f1 HTTP/1.1" 200 None 2026-03-08 01:59:15,777 - lava_callback - DEBUG - Processing node 69aa34e38c4d487a0e7523f1: baseline-arm-baylibre 2026-03-08 01:59:15,777 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa34e38c4d487a0e7523f1 2026-03-08 01:59:15,778 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:59:15,794 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa34e38c4d487a0e7523f1 HTTP/1.1" 200 None 2026-03-08 01:59:15,795 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:59:15,831 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a4d6d27caa0b8e1b59d8ae HTTP/1.1" 200 None 2026-03-08 01:59:15,832 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:59:15,853 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa34e38c4d487a0e7523f1 HTTP/1.1" 200 None 2026-03-08 01:59:15,854 - lava_callback - ERROR - Error processing callback for node 69aa34e38c4d487a0e7523f1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:35886 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48812 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:59:25,109 - lava_callback - INFO - Received callback for node 69accebbbcd207e915d77792 INFO: 172.21.0.1:48816 - "POST /node/69accebbbcd207e915d77792 HTTP/1.0" 202 Accepted 2026-03-08 01:59:25,115 - lava_callback - INFO - Starting async job submit for node 69accebbbcd207e915d77792 2026-03-08 01:59:25,138 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:59:25,152 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accebbbcd207e915d77792 HTTP/1.1" 200 None 2026-03-08 01:59:25,153 - lava_callback - DEBUG - Processing node 69accebbbcd207e915d77792: baseline-arm64-broonie 2026-03-08 01:59:25,436 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:59:25,603 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:59:25,605 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69accebbbcd207e915d77792/log.txt.gz 2026-03-08 01:59:25,765 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:59:25,991 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:59:25,993 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69accebbbcd207e915d77792/lava-logs.yaml 2026-03-08 01:59:26,000 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 01:59:26,169 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 01:59:26,171 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69accebbbcd207e915d77792/lava_callback.json.gz 2026-03-08 01:59:26,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:59:26,208 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accebbbcd207e915d77792 HTTP/1.1" 200 None 2026-03-08 01:59:26,209 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:59:26,225 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 01:59:26,226 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:59:26,257 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accebbbcd207e915d77792 HTTP/1.1" 200 None 2026-03-08 01:59:26,263 - lava_callback - INFO - Completed processing callback for node 69accebbbcd207e915d77792 INFO: 172.21.0.1:48826 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39086 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:59:45,287 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:59:45,308 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:45950 - "GET / HTTP/1.0" 200 OK 2026-03-08 01:59:55,782 - lava_callback - INFO - Received callback for node 69aa350c8c4d487a0e7523ff INFO: 172.21.0.1:39730 - "POST /node/69aa350c8c4d487a0e7523ff HTTP/1.0" 202 Accepted 2026-03-08 01:59:55,787 - lava_callback - INFO - Starting async job submit for node 69aa350c8c4d487a0e7523ff 2026-03-08 01:59:55,788 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:59:55,803 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa350c8c4d487a0e7523ff HTTP/1.1" 200 None 2026-03-08 01:59:55,804 - lava_callback - DEBUG - Processing node 69aa350c8c4d487a0e7523ff: baseline-arm-baylibre 2026-03-08 01:59:55,804 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa350c8c4d487a0e7523ff 2026-03-08 01:59:55,804 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:59:55,818 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa350c8c4d487a0e7523ff HTTP/1.1" 200 None 2026-03-08 01:59:55,820 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:59:55,837 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a4d6d27caa0b8e1b59d8ae HTTP/1.1" 200 None 2026-03-08 01:59:55,840 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 01:59:55,862 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa350c8c4d487a0e7523ff HTTP/1.1" 200 None 2026-03-08 01:59:55,869 - lava_callback - ERROR - Error processing callback for node 69aa350c8c4d487a0e7523ff: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:39740 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40344 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57988 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58000 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42220 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42222 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57822 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42286 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60038 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42872 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54056 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54066 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43276 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43290 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:01:32,907 - lava_callback - INFO - Received callback for node 69acd81bbcd207e915d77cb6 INFO: 172.21.0.1:39610 - "POST /node/69acd81bbcd207e915d77cb6 HTTP/1.0" 202 Accepted 2026-03-08 02:01:32,919 - lava_callback - INFO - Starting async job submit for node 69acd81bbcd207e915d77cb6 2026-03-08 02:01:32,976 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:01:32,995 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd81bbcd207e915d77cb6 HTTP/1.1" 200 None 2026-03-08 02:01:32,996 - lava_callback - DEBUG - Processing node 69acd81bbcd207e915d77cb6: baseline-arm64 2026-03-08 02:01:33,195 - lava_callback - WARNING - Failed to upload log 2026-03-08 02:01:33,296 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:01:33,546 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:01:33,548 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd81bbcd207e915d77cb6/lava-logs.yaml 2026-03-08 02:01:33,552 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:01:33,706 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:01:33,708 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd81bbcd207e915d77cb6/lava_callback.json.gz 2026-03-08 02:01:33,738 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:01:33,753 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd81bbcd207e915d77cb6 HTTP/1.1" 200 None 2026-03-08 02:01:33,755 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:01:33,768 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 02:01:33,770 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:01:33,791 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd81bbcd207e915d77cb6 HTTP/1.1" 200 None 2026-03-08 02:01:33,869 - lava_callback - INFO - Completed processing callback for node 69acd81bbcd207e915d77cb6 INFO: 172.21.0.1:39622 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:01:45,310 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:01:45,326 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:33978 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57666 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59622 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51068 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:51080 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50312 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50326 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51404 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43608 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42954 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56450 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43124 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:43132 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42452 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:03:26,522 - lava_callback - INFO - Received callback for node 69acd7a4bcd207e915d77c8b INFO: 172.21.0.1:42464 - "POST /node/69acd7a4bcd207e915d77c8b HTTP/1.0" 202 Accepted 2026-03-08 02:03:26,529 - lava_callback - INFO - Starting async job submit for node 69acd7a4bcd207e915d77c8b INFO: 172.21.0.1:42468 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:03:26,567 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:03:26,593 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd7a4bcd207e915d77c8b HTTP/1.1" 200 None 2026-03-08 02:03:26,593 - lava_callback - DEBUG - Processing node 69acd7a4bcd207e915d77c8b: baseline-arm64 2026-03-08 02:03:26,983 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:03:27,134 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:03:27,135 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd7a4bcd207e915d77c8b/log.txt.gz 2026-03-08 02:03:27,340 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:03:27,565 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:03:27,567 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd7a4bcd207e915d77c8b/lava-logs.yaml 2026-03-08 02:03:27,576 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:03:27,724 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:03:27,726 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd7a4bcd207e915d77c8b/lava_callback.json.gz DEBUG: Setup failed for 69acd7a4bcd207e915d77c8b. Transitting job node result: incomplete -> fail 2026-03-08 02:03:27,797 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:03:27,812 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd7a4bcd207e915d77c8b HTTP/1.1" 200 None 2026-03-08 02:03:27,814 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:03:27,827 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 02:03:27,828 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:03:27,854 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd7a4bcd207e915d77c8b HTTP/1.1" 200 None 2026-03-08 02:03:27,941 - lava_callback - INFO - Completed processing callback for node 69acd7a4bcd207e915d77c8b INFO: 172.21.0.1:44572 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:03:45,329 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:03:45,344 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:39166 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:03:54,626 - lava_callback - INFO - Received callback for node 69accebdbcd207e915d77798 INFO: 172.21.0.1:40490 - "POST /node/69accebdbcd207e915d77798 HTTP/1.0" 202 Accepted 2026-03-08 02:03:54,632 - lava_callback - INFO - Starting async job submit for node 69accebdbcd207e915d77798 2026-03-08 02:03:54,655 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:03:54,669 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accebdbcd207e915d77798 HTTP/1.1" 200 None 2026-03-08 02:03:54,670 - lava_callback - DEBUG - Processing node 69accebdbcd207e915d77798: baseline-arm64-broonie 2026-03-08 02:03:54,968 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:03:55,118 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:03:55,120 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69accebdbcd207e915d77798/log.txt.gz 2026-03-08 02:03:55,277 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:03:55,483 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:03:55,485 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69accebdbcd207e915d77798/lava-logs.yaml 2026-03-08 02:03:55,491 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:03:55,649 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:03:55,650 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69accebdbcd207e915d77798/lava_callback.json.gz 2026-03-08 02:03:55,677 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:03:55,697 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accebdbcd207e915d77798 HTTP/1.1" 200 None 2026-03-08 02:03:55,699 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:03:55,714 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 02:03:55,716 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:03:55,754 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accebdbcd207e915d77798 HTTP/1.1" 200 None 2026-03-08 02:03:55,755 - lava_callback - INFO - Completed processing callback for node 69accebdbcd207e915d77798 INFO: 172.21.0.1:40496 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57524 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44918 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 02:04:15,346 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:04:15,361 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:44932 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44886 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44898 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41726 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56370 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53564 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46458 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47460 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47468 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38724 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38740 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33094 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59784 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41564 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56098 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52534 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52542 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45624 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45634 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33988 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44338 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34828 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38038 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49110 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:49122 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49926 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49936 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35892 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35438 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52080 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43226 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56306 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56318 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56048 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56056 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60694 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38174 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55870 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55872 - "GET /node/699385fccebe99dd8e070277%5Cn HTTP/1.0" 405 Method Not Allowed INFO: 172.21.0.1:45718 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60752 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:60762 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41128 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41138 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:09:27,075 - lava_callback - INFO - Received callback for node 69acd60dbcd207e915d77bf0 INFO: 172.21.0.1:41152 - "POST /node/69acd60dbcd207e915d77bf0 HTTP/1.0" 202 Accepted 2026-03-08 02:09:27,082 - lava_callback - INFO - Starting async job submit for node 69acd60dbcd207e915d77bf0 2026-03-08 02:09:27,134 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:09:27,160 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd60dbcd207e915d77bf0 HTTP/1.1" 200 None 2026-03-08 02:09:27,161 - lava_callback - DEBUG - Processing node 69acd60dbcd207e915d77bf0: baseline-arm 2026-03-08 02:09:27,737 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:09:27,925 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:09:27,927 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69acd60dbcd207e915d77bf0/log.txt.gz 2026-03-08 02:09:28,261 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:09:28,531 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:09:28,533 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69acd60dbcd207e915d77bf0/lava-logs.yaml 2026-03-08 02:09:28,547 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:09:28,735 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:09:28,737 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69acd60dbcd207e915d77bf0/lava_callback.json.gz DEBUG: Setup failed for 69acd60dbcd207e915d77bf0. Transitting job node result: incomplete -> fail 2026-03-08 02:09:28,850 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:09:28,865 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd60dbcd207e915d77bf0 HTTP/1.1" 200 None 2026-03-08 02:09:28,866 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:09:28,879 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 02:09:28,881 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:09:28,923 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd60dbcd207e915d77bf0 HTTP/1.1" 200 None 2026-03-08 02:09:29,078 - lava_callback - INFO - Completed processing callback for node 69acd60dbcd207e915d77bf0 INFO: 172.21.0.1:40892 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:09:45,364 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:09:45,380 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:43582 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32796 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49914 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54856 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54868 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55484 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55490 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52902 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35158 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50612 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59664 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50822 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:50838 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51170 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51174 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57514 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52240 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:11:50,754 - lava_callback - INFO - Received callback for node 69acd8d0bcd207e915d77cf6 INFO: 172.21.0.1:51886 - "POST /node/69acd8d0bcd207e915d77cf6 HTTP/1.0" 202 Accepted 2026-03-08 02:11:50,759 - lava_callback - INFO - Starting async job submit for node 69acd8d0bcd207e915d77cf6 2026-03-08 02:11:50,763 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:11:50,819 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd8d0bcd207e915d77cf6 HTTP/1.1" 200 None 2026-03-08 02:11:50,819 - lava_callback - DEBUG - Processing node 69acd8d0bcd207e915d77cf6: baseline-arm64-foundriesio 2026-03-08 02:11:50,827 - lava_callback - WARNING - Failed to upload log 2026-03-08 02:11:50,832 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:11:50,979 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:11:50,981 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69acd8d0bcd207e915d77cf6/lava-logs.yaml 2026-03-08 02:11:50,982 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:11:51,115 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:11:51,116 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69acd8d0bcd207e915d77cf6/lava_callback.json.gz 2026-03-08 02:11:51,120 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:11:51,142 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd8d0bcd207e915d77cf6 HTTP/1.1" 200 None 2026-03-08 02:11:51,143 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:11:51,157 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 02:11:51,159 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:11:51,179 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd8d0bcd207e915d77cf6 HTTP/1.1" 200 None 2026-03-08 02:11:51,189 - lava_callback - INFO - Completed processing callback for node 69acd8d0bcd207e915d77cf6 INFO: 172.21.0.1:51894 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:12:06,362 - lava_callback - INFO - Received callback for node 69acd9aabcd207e915d77d4e INFO: 172.21.0.1:50182 - "POST /node/69acd9aabcd207e915d77d4e HTTP/1.0" 202 Accepted 2026-03-08 02:12:06,370 - lava_callback - INFO - Starting async job submit for node 69acd9aabcd207e915d77d4e 2026-03-08 02:12:06,404 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:12:06,418 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd9aabcd207e915d77d4e HTTP/1.1" 200 None 2026-03-08 02:12:06,419 - lava_callback - DEBUG - Processing node 69acd9aabcd207e915d77d4e: baseline-arm64 INFO: 172.21.0.1:50184 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:12:06,788 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:12:06,924 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:12:06,926 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd9aabcd207e915d77d4e/log.txt.gz 2026-03-08 02:12:07,139 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:12:07,356 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:12:07,358 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd9aabcd207e915d77d4e/lava-logs.yaml 2026-03-08 02:12:07,366 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:12:07,541 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:12:07,543 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-69acd9aabcd207e915d77d4e/lava_callback.json.gz DEBUG: Setup failed for 69acd9aabcd207e915d77d4e. Transitting job node result: incomplete -> fail 2026-03-08 02:12:07,613 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:12:07,630 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd9aabcd207e915d77d4e HTTP/1.1" 200 None 2026-03-08 02:12:07,632 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:12:07,646 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 02:12:07,647 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:12:07,675 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd9aabcd207e915d77d4e HTTP/1.1" 200 None 2026-03-08 02:12:07,752 - lava_callback - INFO - Completed processing callback for node 69acd9aabcd207e915d77d4e INFO: 172.21.0.1:36384 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 02:12:15,383 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:12:15,404 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:36392 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35652 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35656 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46652 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:12:43,829 - lava_callback - INFO - Received callback for node 69acd68dbcd207e915d77c37 INFO: 172.21.0.1:52982 - "POST /node/69acd68dbcd207e915d77c37 HTTP/1.0" 202 Accepted 2026-03-08 02:12:43,835 - lava_callback - INFO - Starting async job submit for node 69acd68dbcd207e915d77c37 2026-03-08 02:12:43,866 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:12:43,885 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd68dbcd207e915d77c37 HTTP/1.1" 200 None 2026-03-08 02:12:43,886 - lava_callback - DEBUG - Processing node 69acd68dbcd207e915d77c37: baseline-arm-clabbe 2026-03-08 02:12:44,111 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:12:44,223 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:12:44,225 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acd68dbcd207e915d77c37/log.txt.gz 2026-03-08 02:12:44,341 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:12:44,502 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:12:44,505 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acd68dbcd207e915d77c37/lava-logs.yaml 2026-03-08 02:12:44,509 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:12:44,616 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:12:44,618 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acd68dbcd207e915d77c37/lava_callback.json.gz DEBUG: Setup failed for 69acd68dbcd207e915d77c37. Transitting job node result: incomplete -> fail 2026-03-08 02:12:44,686 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:12:44,701 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd68dbcd207e915d77c37 HTTP/1.1" 200 None 2026-03-08 02:12:44,703 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:12:44,717 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 02:12:44,718 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:12:44,743 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd68dbcd207e915d77c37 HTTP/1.1" 200 None 2026-03-08 02:12:44,806 - lava_callback - INFO - Completed processing callback for node 69acd68dbcd207e915d77c37 2026-03-08 02:12:45,406 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:12:45,422 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:52990 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46180 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38112 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46322 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46332 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36624 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36628 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54282 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36410 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42238 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32810 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55080 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55082 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57922 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57932 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53352 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34088 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53952 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41596 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48870 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48886 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60100 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60110 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35546 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55734 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45524 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54834 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52544 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52548 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55578 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55582 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34336 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58730 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52948 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60220 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53916 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:53924 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39096 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39104 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52774 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51448 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34610 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32972 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38358 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:38364 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35276 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35288 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33878 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38104 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47970 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36996 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57536 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 02:19:15,445 - lava_callback - INFO - Received callback for node 69acd890bcd207e915d77cdd INFO: 172.21.0.1:57548 - "POST /node/69acd890bcd207e915d77cdd HTTP/1.0" 202 Accepted 2026-03-08 02:19:15,451 - lava_callback - INFO - Starting async job submit for node 69acd890bcd207e915d77cdd 2026-03-08 02:19:15,508 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:19:15,523 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd890bcd207e915d77cdd HTTP/1.1" 200 None 2026-03-08 02:19:15,524 - lava_callback - DEBUG - Processing node 69acd890bcd207e915d77cdd: ltp-smoke 2026-03-08 02:19:16,006 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:19:16,154 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:19:16,156 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69acd890bcd207e915d77cdd/log.txt.gz 2026-03-08 02:19:16,448 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:57562 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:19:16,670 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:19:16,672 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69acd890bcd207e915d77cdd/lava-logs.yaml 2026-03-08 02:19:16,684 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:19:16,881 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:19:16,882 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69acd890bcd207e915d77cdd/lava_callback.json.gz 2026-03-08 02:19:17,000 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:19:17,014 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd890bcd207e915d77cdd HTTP/1.1" 200 None 2026-03-08 02:19:17,015 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:19:17,029 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc012bcd207e915d76b9c HTTP/1.1" 200 None 2026-03-08 02:19:17,030 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:19:17,055 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd890bcd207e915d77cdd HTTP/1.1" 200 None 2026-03-08 02:19:17,166 - lava_callback - INFO - Completed processing callback for node 69acd890bcd207e915d77cdd INFO: 172.21.0.1:54892 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54906 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57088 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:19:36,932 - lava_callback - INFO - Received callback for node 69aa39a48c4d487a0e75263c INFO: 172.21.0.1:57096 - "POST /node/69aa39a48c4d487a0e75263c HTTP/1.0" 202 Accepted 2026-03-08 02:19:36,937 - lava_callback - INFO - Starting async job submit for node 69aa39a48c4d487a0e75263c 2026-03-08 02:19:36,938 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:19:36,961 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa39a48c4d487a0e75263c HTTP/1.1" 200 None 2026-03-08 02:19:36,962 - lava_callback - DEBUG - Processing node 69aa39a48c4d487a0e75263c: baseline-arm64-baylibre 2026-03-08 02:19:36,962 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa39a48c4d487a0e75263c 2026-03-08 02:19:36,963 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:19:36,987 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa39a48c4d487a0e75263c HTTP/1.1" 200 None 2026-03-08 02:19:36,988 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:19:37,009 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a4d6e67caa0b8e1b59d8c1 HTTP/1.1" 200 None 2026-03-08 02:19:37,011 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:19:37,057 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa39a48c4d487a0e75263c HTTP/1.1" 200 None 2026-03-08 02:19:37,059 - lava_callback - ERROR - Error processing callback for node 69aa39a48c4d487a0e75263c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-08 02:19:45,425 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:19:45,447 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:47910 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51446 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:19:56,833 - lava_callback - INFO - Received callback for node 69acceb5bcd207e915d77785 INFO: 172.21.0.1:51462 - "POST /node/69acceb5bcd207e915d77785 HTTP/1.0" 202 Accepted 2026-03-08 02:19:56,839 - lava_callback - INFO - Starting async job submit for node 69acceb5bcd207e915d77785 2026-03-08 02:19:56,861 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:19:56,875 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acceb5bcd207e915d77785 HTTP/1.1" 200 None 2026-03-08 02:19:56,876 - lava_callback - DEBUG - Processing node 69acceb5bcd207e915d77785: baseline-arm64-broonie 2026-03-08 02:19:57,086 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:19:57,230 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:19:57,232 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69acceb5bcd207e915d77785/log.txt.gz 2026-03-08 02:19:57,372 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:19:57,581 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:19:57,583 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69acceb5bcd207e915d77785/lava-logs.yaml 2026-03-08 02:19:57,589 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:19:57,731 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:19:57,732 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69acceb5bcd207e915d77785/lava_callback.json.gz 2026-03-08 02:19:57,755 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:19:57,770 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acceb5bcd207e915d77785 HTTP/1.1" 200 None 2026-03-08 02:19:57,771 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:19:57,787 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 02:19:57,788 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:19:57,813 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acceb5bcd207e915d77785 HTTP/1.1" 200 None 2026-03-08 02:19:57,814 - lava_callback - INFO - Completed processing callback for node 69acceb5bcd207e915d77785 INFO: 172.21.0.1:36814 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38000 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 02:20:15,450 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:20:15,465 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:38002 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59634 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59642 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41292 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36652 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53860 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47248 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54616 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54630 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:21:16,525 - lava_callback - INFO - Received callback for node 69acdb02bcd207e915d77da3 INFO: 172.21.0.1:54636 - "POST /node/69acdb02bcd207e915d77da3 HTTP/1.0" 202 Accepted 2026-03-08 02:21:16,534 - lava_callback - INFO - Starting async job submit for node 69acdb02bcd207e915d77da3 2026-03-08 02:21:16,538 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:21:16,556 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acdb02bcd207e915d77da3 HTTP/1.1" 200 None 2026-03-08 02:21:16,557 - lava_callback - DEBUG - Processing node 69acdb02bcd207e915d77da3: baseline-arm64-foundriesio 2026-03-08 02:21:16,566 - lava_callback - WARNING - Failed to upload log 2026-03-08 02:21:16,572 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:21:16,752 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:21:16,754 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69acdb02bcd207e915d77da3/lava-logs.yaml 2026-03-08 02:21:16,756 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:21:16,897 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:21:16,898 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-foundriesio-69acdb02bcd207e915d77da3/lava_callback.json.gz 2026-03-08 02:21:16,902 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:21:16,936 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acdb02bcd207e915d77da3 HTTP/1.1" 200 None 2026-03-08 02:21:16,937 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:21:16,952 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 02:21:16,953 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:21:17,019 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acdb02bcd207e915d77da3 HTTP/1.1" 200 None 2026-03-08 02:21:17,025 - lava_callback - INFO - Completed processing callback for node 69acdb02bcd207e915d77da3 INFO: 172.21.0.1:58856 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58872 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53360 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:21:45,467 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:21:45,485 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:59912 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51072 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46342 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51094 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:51100 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36254 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36260 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58472 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42772 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60134 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51562 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54700 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54712 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59354 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59356 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49338 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34532 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41772 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57084 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51522 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:51538 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53050 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53056 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47288 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35764 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51876 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:24:57,208 - lava_callback - INFO - Received callback for node 69aa3aee8c4d487a0e7526dd INFO: 172.21.0.1:51892 - "POST /node/69aa3aee8c4d487a0e7526dd HTTP/1.0" 202 Accepted 2026-03-08 02:24:57,214 - lava_callback - INFO - Starting async job submit for node 69aa3aee8c4d487a0e7526dd 2026-03-08 02:24:57,215 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:24:57,229 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa3aee8c4d487a0e7526dd HTTP/1.1" 200 None 2026-03-08 02:24:57,230 - lava_callback - DEBUG - Processing node 69aa3aee8c4d487a0e7526dd: baseline-arm64-baylibre 2026-03-08 02:24:57,230 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa3aee8c4d487a0e7526dd 2026-03-08 02:24:57,231 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:24:57,245 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa3aee8c4d487a0e7526dd HTTP/1.1" 200 None 2026-03-08 02:24:57,246 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:24:57,265 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a2341790d011217c9f7d18 HTTP/1.1" 200 None 2026-03-08 02:24:57,267 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:24:57,327 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa3aee8c4d487a0e7526dd HTTP/1.1" 200 None 2026-03-08 02:24:57,328 - lava_callback - ERROR - Error processing callback for node 69aa3aee8c4d487a0e7526dd: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:42992 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50676 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:50678 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34480 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34494 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59072 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48876 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44764 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:26:00,492 - lava_callback - INFO - Received callback for node 69acda59bcd207e915d77d71 INFO: 172.21.0.1:54258 - "POST /node/69acda59bcd207e915d77d71 HTTP/1.0" 202 Accepted 2026-03-08 02:26:00,499 - lava_callback - INFO - Starting async job submit for node 69acda59bcd207e915d77d71 2026-03-08 02:26:00,551 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:26:00,565 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acda59bcd207e915d77d71 HTTP/1.1" 200 None 2026-03-08 02:26:00,566 - lava_callback - DEBUG - Processing node 69acda59bcd207e915d77d71: baseline-arm 2026-03-08 02:26:01,152 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:26:01,295 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:26:01,297 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-69acda59bcd207e915d77d71/log.txt.gz 2026-03-08 02:26:01,633 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:26:01,879 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:26:01,882 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-69acda59bcd207e915d77d71/lava-logs.yaml 2026-03-08 02:26:01,896 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:26:02,055 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:26:02,057 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-69acda59bcd207e915d77d71/lava_callback.json.gz DEBUG: Setup failed for 69acda59bcd207e915d77d71. Transitting job node result: incomplete -> fail 2026-03-08 02:26:02,166 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:26:02,181 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acda59bcd207e915d77d71 HTTP/1.1" 200 None 2026-03-08 02:26:02,182 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:26:02,221 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 02:26:02,223 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:26:02,247 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acda59bcd207e915d77d71 HTTP/1.1" 200 None 2026-03-08 02:26:02,345 - lava_callback - INFO - Completed processing callback for node 69acda59bcd207e915d77d71 INFO: 172.21.0.1:54262 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47650 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 02:26:15,489 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:26:15,505 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:47658 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:26:17,295 - lava_callback - INFO - Received callback for node 69aa3b3f8c4d487a0e752706 INFO: 172.21.0.1:47668 - "POST /node/69aa3b3f8c4d487a0e752706 HTTP/1.0" 202 Accepted 2026-03-08 02:26:17,300 - lava_callback - INFO - Starting async job submit for node 69aa3b3f8c4d487a0e752706 2026-03-08 02:26:17,301 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:26:17,339 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa3b3f8c4d487a0e752706 HTTP/1.1" 200 None 2026-03-08 02:26:17,340 - lava_callback - DEBUG - Processing node 69aa3b3f8c4d487a0e752706: baseline-arm64-baylibre 2026-03-08 02:26:17,340 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa3b3f8c4d487a0e752706 2026-03-08 02:26:17,341 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:26:17,456 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa3b3f8c4d487a0e752706 HTTP/1.1" 200 None 2026-03-08 02:26:17,458 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:26:17,471 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a2341790d011217c9f7d18 HTTP/1.1" 200 None 2026-03-08 02:26:17,472 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:26:17,492 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa3b3f8c4d487a0e752706 HTTP/1.1" 200 None 2026-03-08 02:26:17,493 - lava_callback - ERROR - Error processing callback for node 69aa3b3f8c4d487a0e752706: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:45554 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45564 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41720 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49110 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49430 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46904 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51386 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:51398 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34204 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34220 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44788 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44876 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49846 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36348 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56864 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56878 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:28:17,396 - lava_callback - INFO - Received callback for node 69aa3bba8c4d487a0e752740 INFO: 172.21.0.1:56880 - "POST /node/69aa3bba8c4d487a0e752740 HTTP/1.0" 202 Accepted 2026-03-08 02:28:17,401 - lava_callback - INFO - Starting async job submit for node 69aa3bba8c4d487a0e752740 2026-03-08 02:28:17,402 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:28:17,439 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa3bba8c4d487a0e752740 HTTP/1.1" 200 None 2026-03-08 02:28:17,440 - lava_callback - DEBUG - Processing node 69aa3bba8c4d487a0e752740: baseline-arm64-baylibre 2026-03-08 02:28:17,440 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa3bba8c4d487a0e752740 2026-03-08 02:28:17,441 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:28:17,455 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa3bba8c4d487a0e752740 HTTP/1.1" 200 None 2026-03-08 02:28:17,456 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:28:17,469 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a2341790d011217c9f7d18 HTTP/1.1" 200 None 2026-03-08 02:28:17,471 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:28:17,490 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa3bba8c4d487a0e752740 HTTP/1.1" 200 None 2026-03-08 02:28:17,496 - lava_callback - ERROR - Error processing callback for node 69aa3bba8c4d487a0e752740: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:35536 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35540 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45416 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48220 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47698 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56370 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33588 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:33604 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49730 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49734 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55674 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37610 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38922 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:30:00,142 - lava_callback - INFO - Received callback for node 69acd6b4bcd207e915d77c3f INFO: 172.21.0.1:48036 - "POST /node/69acd6b4bcd207e915d77c3f HTTP/1.0" 202 Accepted 2026-03-08 02:30:00,150 - lava_callback - INFO - Starting async job submit for node 69acd6b4bcd207e915d77c3f 2026-03-08 02:30:00,194 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:30:00,211 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd6b4bcd207e915d77c3f HTTP/1.1" 200 None 2026-03-08 02:30:00,212 - lava_callback - DEBUG - Processing node 69acd6b4bcd207e915d77c3f: baseline-nfs-arm64 2026-03-08 02:30:01,312 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:30:01,537 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:30:01,539 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acd6b4bcd207e915d77c3f/log.txt.gz 2026-03-08 02:30:02,159 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:30:02,453 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:30:02,456 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acd6b4bcd207e915d77c3f/lava-logs.yaml 2026-03-08 02:30:02,477 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:30:02,681 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:30:02,682 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acd6b4bcd207e915d77c3f/lava_callback.json.gz DEBUG: Setup failed for 69acd6b4bcd207e915d77c3f. Transitting job node result: incomplete -> fail 2026-03-08 02:30:02,770 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:30:02,785 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd6b4bcd207e915d77c3f HTTP/1.1" 200 None 2026-03-08 02:30:02,787 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:30:02,806 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 02:30:02,807 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:30:02,840 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd6b4bcd207e915d77c3f HTTP/1.1" 200 None 2026-03-08 02:30:02,932 - lava_callback - INFO - Completed processing callback for node 69acd6b4bcd207e915d77c3f INFO: 172.21.0.1:48040 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55620 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 02:30:15,508 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:30:15,523 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:55632 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34208 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34220 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38828 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43538 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45218 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50392 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53466 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:53480 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43028 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43044 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34588 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:31:37,576 - lava_callback - INFO - Received callback for node 69aa3c868c4d487a0e7527b3 INFO: 172.21.0.1:34592 - "POST /node/69aa3c868c4d487a0e7527b3 HTTP/1.0" 202 Accepted 2026-03-08 02:31:37,582 - lava_callback - INFO - Starting async job submit for node 69aa3c868c4d487a0e7527b3 2026-03-08 02:31:37,582 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:31:37,597 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa3c868c4d487a0e7527b3 HTTP/1.1" 200 None 2026-03-08 02:31:37,598 - lava_callback - DEBUG - Processing node 69aa3c868c4d487a0e7527b3: baseline-arm64-baylibre 2026-03-08 02:31:37,598 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa3c868c4d487a0e7527b3 2026-03-08 02:31:37,599 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:31:37,612 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa3c868c4d487a0e7527b3 HTTP/1.1" 200 None 2026-03-08 02:31:37,613 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:31:37,627 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a2341790d011217c9f7d18 HTTP/1.1" 200 None 2026-03-08 02:31:37,628 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:31:37,650 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa3c868c4d487a0e7527b3 HTTP/1.1" 200 None 2026-03-08 02:31:37,651 - lava_callback - ERROR - Error processing callback for node 69aa3c868c4d487a0e7527b3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:58102 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45414 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39334 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46848 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46850 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58014 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58024 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36538 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:32:44,817 - lava_callback - INFO - Received callback for node 69acdb3ebcd207e915d77db7 INFO: 172.21.0.1:40464 - "POST /node/69acdb3ebcd207e915d77db7 HTTP/1.0" 202 Accepted 2026-03-08 02:32:44,823 - lava_callback - INFO - Starting async job submit for node 69acdb3ebcd207e915d77db7 2026-03-08 02:32:44,853 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:32:44,876 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acdb3ebcd207e915d77db7 HTTP/1.1" 200 None 2026-03-08 02:32:44,877 - lava_callback - DEBUG - Processing node 69acdb3ebcd207e915d77db7: baseline-arm-clabbe 2026-03-08 02:32:45,091 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:32:45,238 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:32:45,240 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acdb3ebcd207e915d77db7/log.txt.gz 2026-03-08 02:32:45,358 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:32:45,555 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:32:45,558 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acdb3ebcd207e915d77db7/lava-logs.yaml 2026-03-08 02:32:45,562 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:32:45,689 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:32:45,690 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-clabbe-69acdb3ebcd207e915d77db7/lava_callback.json.gz DEBUG: Setup failed for 69acdb3ebcd207e915d77db7. Transitting job node result: incomplete -> fail 2026-03-08 02:32:45,757 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:32:45,776 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acdb3ebcd207e915d77db7 HTTP/1.1" 200 None 2026-03-08 02:32:45,778 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:32:45,791 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 02:32:45,792 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:32:45,816 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acdb3ebcd207e915d77db7 HTTP/1.1" 200 None 2026-03-08 02:32:45,886 - lava_callback - INFO - Completed processing callback for node 69acdb3ebcd207e915d77db7 INFO: 172.21.0.1:40468 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49908 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44452 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45268 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 02:33:15,526 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:33:15,564 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:45278 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41730 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41742 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36024 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51764 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36926 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:33:57,645 - lava_callback - INFO - Received callback for node 69aa3d038c4d487a0e752800 INFO: 172.21.0.1:36928 - "POST /node/69aa3d038c4d487a0e752800 HTTP/1.0" 202 Accepted 2026-03-08 02:33:57,651 - lava_callback - INFO - Starting async job submit for node 69aa3d038c4d487a0e752800 2026-03-08 02:33:57,652 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:33:57,667 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa3d038c4d487a0e752800 HTTP/1.1" 200 None 2026-03-08 02:33:57,668 - lava_callback - DEBUG - Processing node 69aa3d038c4d487a0e752800: baseline-arm64-baylibre 2026-03-08 02:33:57,668 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa3d038c4d487a0e752800 2026-03-08 02:33:57,669 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:33:57,686 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa3d038c4d487a0e752800 HTTP/1.1" 200 None 2026-03-08 02:33:57,688 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:33:57,701 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a2341790d011217c9f7d18 HTTP/1.1" 200 None 2026-03-08 02:33:57,703 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:33:57,724 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa3d038c4d487a0e752800 HTTP/1.1" 200 None 2026-03-08 02:33:57,730 - lava_callback - ERROR - Error processing callback for node 69aa3d038c4d487a0e752800: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:50178 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42928 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:42940 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56578 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56594 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34886 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38972 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51266 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32850 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40258 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:40268 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60284 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60292 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43018 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60758 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54812 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54134 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33320 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:33324 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43996 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44006 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33070 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43160 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55038 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:36:58,049 - lava_callback - INFO - Received callback for node 69acd7f9bcd207e915d77ca5 INFO: 172.21.0.1:55044 - "POST /node/69acd7f9bcd207e915d77ca5 HTTP/1.0" 202 Accepted 2026-03-08 02:36:58,097 - lava_callback - INFO - Starting async job submit for node 69acd7f9bcd207e915d77ca5 2026-03-08 02:36:58,121 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:36:58,138 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd7f9bcd207e915d77ca5 HTTP/1.1" 200 None 2026-03-08 02:36:58,139 - lava_callback - DEBUG - Processing node 69acd7f9bcd207e915d77ca5: baseline-arm-pengutronix 2026-03-08 02:36:58,242 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:36:58,393 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:36:58,395 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69acd7f9bcd207e915d77ca5/log.txt.gz 2026-03-08 02:36:58,453 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:36:58,673 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:36:58,675 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69acd7f9bcd207e915d77ca5/lava-logs.yaml 2026-03-08 02:36:58,678 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:36:58,820 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:36:58,822 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69acd7f9bcd207e915d77ca5/lava_callback.json.gz DEBUG: Setup failed for 69acd7f9bcd207e915d77ca5. Transitting job node result: incomplete -> fail 2026-03-08 02:36:58,876 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:36:58,891 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acd7f9bcd207e915d77ca5 HTTP/1.1" 200 None 2026-03-08 02:36:58,893 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:36:58,907 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 02:36:58,909 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:36:58,932 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acd7f9bcd207e915d77ca5 HTTP/1.1" 200 None 2026-03-08 02:36:58,995 - lava_callback - INFO - Completed processing callback for node 69acd7f9bcd207e915d77ca5 INFO: 172.21.0.1:45106 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49616 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 02:37:15,567 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:37:15,583 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:49624 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:37:17,838 - lava_callback - INFO - Received callback for node 69aa3dca8c4d487a0e75285c INFO: 172.21.0.1:49626 - "POST /node/69aa3dca8c4d487a0e75285c HTTP/1.0" 202 Accepted 2026-03-08 02:37:17,843 - lava_callback - INFO - Starting async job submit for node 69aa3dca8c4d487a0e75285c 2026-03-08 02:37:17,844 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:37:17,859 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa3dca8c4d487a0e75285c HTTP/1.1" 200 None 2026-03-08 02:37:17,859 - lava_callback - DEBUG - Processing node 69aa3dca8c4d487a0e75285c: baseline-arm64-baylibre 2026-03-08 02:37:17,860 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa3dca8c4d487a0e75285c 2026-03-08 02:37:17,860 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:37:17,874 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa3dca8c4d487a0e75285c HTTP/1.1" 200 None 2026-03-08 02:37:17,875 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:37:17,901 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a2341790d011217c9f7d18 HTTP/1.1" 200 None 2026-03-08 02:37:17,903 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:37:17,923 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa3dca8c4d487a0e75285c HTTP/1.1" 200 None 2026-03-08 02:37:17,943 - lava_callback - ERROR - Error processing callback for node 69aa3dca8c4d487a0e75285c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:54020 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54036 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35094 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46098 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40072 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45372 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34602 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34604 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57210 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57222 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39684 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39034 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40608 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35210 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40392 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:40398 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43630 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43640 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43930 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48484 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43432 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46472 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33454 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:33458 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41144 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41158 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39412 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57226 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35490 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53214 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59276 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59278 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59010 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59022 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:41:26,656 - lava_callback - INFO - Received callback for node 69acceb9bcd207e915d7778e INFO: 172.21.0.1:59024 - "POST /node/69acceb9bcd207e915d7778e HTTP/1.0" 202 Accepted 2026-03-08 02:41:26,662 - lava_callback - INFO - Starting async job submit for node 69acceb9bcd207e915d7778e 2026-03-08 02:41:26,683 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:41:26,699 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acceb9bcd207e915d7778e HTTP/1.1" 200 None 2026-03-08 02:41:26,700 - lava_callback - DEBUG - Processing node 69acceb9bcd207e915d7778e: baseline-arm64-broonie 2026-03-08 02:41:26,959 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:41:27,103 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:41:27,105 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69acceb9bcd207e915d7778e/log.txt.gz 2026-03-08 02:41:27,291 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:41:27,509 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:41:27,511 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69acceb9bcd207e915d7778e/lava-logs.yaml 2026-03-08 02:41:27,518 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:41:27,671 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:41:27,673 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69acceb9bcd207e915d7778e/lava_callback.json.gz 2026-03-08 02:41:27,696 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:41:27,711 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acceb9bcd207e915d7778e HTTP/1.1" 200 None 2026-03-08 02:41:27,713 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:41:27,726 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 02:41:27,728 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:41:27,752 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acceb9bcd207e915d7778e HTTP/1.1" 200 None 2026-03-08 02:41:27,759 - lava_callback - INFO - Completed processing callback for node 69acceb9bcd207e915d7778e INFO: 172.21.0.1:49516 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:41:45,585 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:41:45,612 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:59246 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49492 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45992 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58486 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 02:42:16,319 - lava_callback - INFO - Received callback for node 69acceb6bcd207e915d77788 INFO: 172.21.0.1:58498 - "POST /node/69acceb6bcd207e915d77788 HTTP/1.0" 202 Accepted 2026-03-08 02:42:16,325 - lava_callback - INFO - Starting async job submit for node 69acceb6bcd207e915d77788 2026-03-08 02:42:16,347 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:42:16,361 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acceb6bcd207e915d77788 HTTP/1.1" 200 None 2026-03-08 02:42:16,362 - lava_callback - DEBUG - Processing node 69acceb6bcd207e915d77788: baseline-arm64-broonie INFO: 172.21.0.1:58504 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:42:16,589 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:42:16,708 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:42:16,710 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69acceb6bcd207e915d77788/log.txt.gz 2026-03-08 02:42:16,843 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:42:17,071 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:42:17,074 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69acceb6bcd207e915d77788/lava-logs.yaml 2026-03-08 02:42:17,080 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:42:17,236 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:42:17,238 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69acceb6bcd207e915d77788/lava_callback.json.gz 2026-03-08 02:42:17,262 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:42:17,276 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acceb6bcd207e915d77788 HTTP/1.1" 200 None 2026-03-08 02:42:17,278 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:42:17,291 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 02:42:17,293 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:42:17,318 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acceb6bcd207e915d77788 HTTP/1.1" 200 None 2026-03-08 02:42:17,328 - lava_callback - INFO - Completed processing callback for node 69acceb6bcd207e915d77788 INFO: 172.21.0.1:33438 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33444 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33078 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:42:45,615 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:42:45,632 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:45116 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35442 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45916 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51750 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:51758 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37970 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37974 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56658 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58396 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39422 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58360 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54594 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54604 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53252 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53258 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44832 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38782 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40190 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47990 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37522 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:37526 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37000 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37010 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48324 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45638 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45566 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37634 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60780 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:60788 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52010 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52022 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:46:26,864 - lava_callback - INFO - Received callback for node 69accebfbcd207e915d7779b INFO: 172.21.0.1:52032 - "POST /node/69accebfbcd207e915d7779b HTTP/1.0" 202 Accepted 2026-03-08 02:46:26,871 - lava_callback - INFO - Starting async job submit for node 69accebfbcd207e915d7779b 2026-03-08 02:46:26,893 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:46:26,907 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accebfbcd207e915d7779b HTTP/1.1" 200 None 2026-03-08 02:46:26,908 - lava_callback - DEBUG - Processing node 69accebfbcd207e915d7779b: baseline-arm64-broonie 2026-03-08 02:46:27,257 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:46:27,400 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:46:27,402 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69accebfbcd207e915d7779b/log.txt.gz 2026-03-08 02:46:27,624 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:46:27,844 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:46:27,846 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69accebfbcd207e915d7779b/lava-logs.yaml 2026-03-08 02:46:27,854 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:46:27,994 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:46:27,996 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-broonie-69accebfbcd207e915d7779b/lava_callback.json.gz 2026-03-08 02:46:28,019 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:46:28,033 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69accebfbcd207e915d7779b HTTP/1.1" 200 None 2026-03-08 02:46:28,034 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:46:28,047 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 02:46:28,049 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:46:28,081 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69accebfbcd207e915d7779b HTTP/1.1" 200 None 2026-03-08 02:46:28,088 - lava_callback - INFO - Completed processing callback for node 69accebfbcd207e915d7779b INFO: 172.21.0.1:34128 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:46:45,635 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:46:45,654 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:36150 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54244 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40770 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41118 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:41122 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58714 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58730 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41556 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45020 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40446 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51958 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35654 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:35658 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44252 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44260 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33920 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46092 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54772 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38394 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56240 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56256 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37374 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37380 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38438 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36846 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53346 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58582 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34820 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34824 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:50:19,814 - lava_callback - INFO - Received callback for node 69acdf34bcd207e915d77eb2 INFO: 172.21.0.1:38362 - "POST /node/69acdf34bcd207e915d77eb2 HTTP/1.0" 202 Accepted 2026-03-08 02:50:19,822 - lava_callback - INFO - Starting async job submit for node 69acdf34bcd207e915d77eb2 2026-03-08 02:50:19,866 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:50:19,881 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acdf34bcd207e915d77eb2 HTTP/1.1" 200 None 2026-03-08 02:50:19,882 - lava_callback - DEBUG - Processing node 69acdf34bcd207e915d77eb2: baseline-nfs-arm64 2026-03-08 02:50:20,985 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:50:21,183 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:50:21,185 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acdf34bcd207e915d77eb2/log.txt.gz 2026-03-08 02:50:21,801 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:38368 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:50:22,027 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:50:22,029 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acdf34bcd207e915d77eb2/lava-logs.yaml 2026-03-08 02:50:22,051 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:50:22,244 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:50:22,246 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69acdf34bcd207e915d77eb2/lava_callback.json.gz DEBUG: Setup failed for 69acdf34bcd207e915d77eb2. Transitting job node result: incomplete -> fail 2026-03-08 02:50:22,341 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:50:22,361 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acdf34bcd207e915d77eb2 HTTP/1.1" 200 None 2026-03-08 02:50:22,363 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:50:22,377 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 02:50:22,378 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:50:22,406 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acdf34bcd207e915d77eb2 HTTP/1.1" 200 None 2026-03-08 02:50:22,544 - lava_callback - INFO - Completed processing callback for node 69acdf34bcd207e915d77eb2 INFO: 172.21.0.1:38378 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55050 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:50:45,656 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:50:45,676 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:49968 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41688 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33478 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59388 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59396 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41030 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41034 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37264 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36580 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59574 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49648 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58274 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58284 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60374 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60386 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42948 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49550 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58398 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53456 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54558 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54572 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58356 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58370 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53474 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38466 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44400 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53816 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57368 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:57376 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46238 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46252 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56828 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40950 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33698 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50724 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44844 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44852 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53064 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53068 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38044 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57620 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50300 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40458 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53522 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:53526 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46946 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46962 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34318 - "GET / HTTP/1.0" 200 OK 2026-03-08 02:56:38,601 - lava_callback - INFO - Received callback for node 69ace0d9bcd207e915d77f26 INFO: 172.21.0.1:34330 - "POST /node/69ace0d9bcd207e915d77f26 HTTP/1.0" 202 Accepted 2026-03-08 02:56:38,649 - lava_callback - INFO - Starting async job submit for node 69ace0d9bcd207e915d77f26 2026-03-08 02:56:38,683 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:56:38,700 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ace0d9bcd207e915d77f26 HTTP/1.1" 200 None 2026-03-08 02:56:38,701 - lava_callback - DEBUG - Processing node 69ace0d9bcd207e915d77f26: baseline-arm-pengutronix 2026-03-08 02:56:38,807 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:56:38,933 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:56:38,934 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69ace0d9bcd207e915d77f26/log.txt.gz 2026-03-08 02:56:39,000 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:56:39,209 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:56:39,211 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69ace0d9bcd207e915d77f26/lava-logs.yaml 2026-03-08 02:56:39,214 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 02:56:39,347 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 02:56:39,348 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-69ace0d9bcd207e915d77f26/lava_callback.json.gz DEBUG: Setup failed for 69ace0d9bcd207e915d77f26. Transitting job node result: incomplete -> fail 2026-03-08 02:56:39,398 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:56:39,413 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ace0d9bcd207e915d77f26 HTTP/1.1" 200 None 2026-03-08 02:56:39,415 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:56:39,430 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 02:56:39,432 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:56:39,458 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69ace0d9bcd207e915d77f26 HTTP/1.1" 200 None 2026-03-08 02:56:39,539 - lava_callback - INFO - Completed processing callback for node 69ace0d9bcd207e915d77f26 2026-03-08 02:56:45,679 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 02:56:45,697 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:52684 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37250 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58600 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34462 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34464 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49944 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49958 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33646 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59034 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50370 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60010 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59040 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59054 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59728 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59732 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55086 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45636 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48350 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41084 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32994 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:32996 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34668 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34684 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35454 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58132 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49108 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53816 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57818 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:57832 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36552 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36568 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55038 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56664 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45614 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40322 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57142 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:57146 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33152 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33164 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46868 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44702 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57456 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48742 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38040 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:38050 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33904 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33908 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60948 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44990 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46120 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:03:00,080 - lava_callback - INFO - Received callback for node 69aa43d28c4d487a0e752a7e INFO: 172.21.0.1:54802 - "POST /node/69aa43d28c4d487a0e752a7e HTTP/1.0" 202 Accepted 2026-03-08 03:03:00,086 - lava_callback - INFO - Starting async job submit for node 69aa43d28c4d487a0e752a7e 2026-03-08 03:03:00,087 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:03:00,104 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa43d28c4d487a0e752a7e HTTP/1.1" 200 None 2026-03-08 03:03:00,104 - lava_callback - DEBUG - Processing node 69aa43d28c4d487a0e752a7e: baseline-arm-clabbe 2026-03-08 03:03:00,105 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa43d28c4d487a0e752a7e 2026-03-08 03:03:00,105 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:03:00,121 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa43d28c4d487a0e752a7e HTTP/1.1" 200 None 2026-03-08 03:03:00,122 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:03:00,142 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a93c6a7ee78ae610caa67c HTTP/1.1" 200 None 2026-03-08 03:03:00,144 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:03:00,168 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa43d28c4d487a0e752a7e HTTP/1.1" 200 None 2026-03-08 03:03:00,173 - lava_callback - ERROR - Error processing callback for node 69aa43d28c4d487a0e752a7e: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:54806 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52564 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52574 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40924 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40938 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45754 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:03:39,863 - lava_callback - INFO - Received callback for node 69aa43fd8c4d487a0e752aa4 INFO: 172.21.0.1:55986 - "POST /node/69aa43fd8c4d487a0e752aa4 HTTP/1.0" 202 Accepted 2026-03-08 03:03:39,869 - lava_callback - INFO - Starting async job submit for node 69aa43fd8c4d487a0e752aa4 2026-03-08 03:03:39,869 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:03:39,884 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa43fd8c4d487a0e752aa4 HTTP/1.1" 200 None 2026-03-08 03:03:39,885 - lava_callback - DEBUG - Processing node 69aa43fd8c4d487a0e752aa4: baseline-arm-clabbe 2026-03-08 03:03:39,885 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa43fd8c4d487a0e752aa4 2026-03-08 03:03:39,885 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:03:39,898 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa43fd8c4d487a0e752aa4 HTTP/1.1" 200 None 2026-03-08 03:03:39,900 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:03:39,916 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a93c6a7ee78ae610caa67c HTTP/1.1" 200 None 2026-03-08 03:03:39,918 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:03:39,938 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa43fd8c4d487a0e752aa4 HTTP/1.1" 200 None 2026-03-08 03:03:39,939 - lava_callback - ERROR - Error processing callback for node 69aa43fd8c4d487a0e752aa4: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:55988 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46322 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53176 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58830 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58832 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45046 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45062 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59312 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38068 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50654 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59944 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46122 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46126 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52662 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52678 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59376 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52242 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45726 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36288 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36894 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:36906 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60332 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60348 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43834 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60064 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48720 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40072 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42008 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:42024 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58660 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58668 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59924 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42582 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54734 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60950 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37924 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:37936 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44342 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44354 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57842 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58414 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33756 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:08:59,515 - lava_callback - INFO - Received callback for node 69aa453b8c4d487a0e752b53 INFO: 172.21.0.1:33758 - "POST /node/69aa453b8c4d487a0e752b53 HTTP/1.0" 202 Accepted 2026-03-08 03:08:59,523 - lava_callback - INFO - Starting async job submit for node 69aa453b8c4d487a0e752b53 2026-03-08 03:08:59,524 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:08:59,557 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa453b8c4d487a0e752b53 HTTP/1.1" 200 None 2026-03-08 03:08:59,558 - lava_callback - DEBUG - Processing node 69aa453b8c4d487a0e752b53: baseline-arm-baylibre 2026-03-08 03:08:59,558 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa453b8c4d487a0e752b53 2026-03-08 03:08:59,559 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:08:59,573 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa453b8c4d487a0e752b53 HTTP/1.1" 200 None 2026-03-08 03:08:59,575 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:08:59,597 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6ed9c2da7ccec10683e47 HTTP/1.1" 200 None 2026-03-08 03:08:59,598 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:08:59,620 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa453b8c4d487a0e752b53 HTTP/1.1" 200 None 2026-03-08 03:08:59,625 - lava_callback - ERROR - Error processing callback for node 69aa453b8c4d487a0e752b53: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:32822 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49896 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:49898 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43384 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43398 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51954 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49122 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45268 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39494 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55692 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55706 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34120 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34136 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46328 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:10:41,484 - lava_callback - INFO - Received callback for node 69ace3f7bcd207e915d77ffc INFO: 172.21.0.1:59658 - "POST /node/69ace3f7bcd207e915d77ffc HTTP/1.0" 202 Accepted 2026-03-08 03:10:41,492 - lava_callback - INFO - Starting async job submit for node 69ace3f7bcd207e915d77ffc 2026-03-08 03:10:41,534 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:10:41,549 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ace3f7bcd207e915d77ffc HTTP/1.1" 200 None 2026-03-08 03:10:41,550 - lava_callback - DEBUG - Processing node 69ace3f7bcd207e915d77ffc: baseline-nfs-arm64 2026-03-08 03:10:42,622 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:10:42,820 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:10:42,822 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69ace3f7bcd207e915d77ffc/log.txt.gz 2026-03-08 03:10:43,463 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:10:43,719 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:10:43,722 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69ace3f7bcd207e915d77ffc/lava-logs.yaml 2026-03-08 03:10:43,743 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:10:43,954 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:10:43,956 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-nfs-arm64-69ace3f7bcd207e915d77ffc/lava_callback.json.gz DEBUG: Setup failed for 69ace3f7bcd207e915d77ffc. Transitting job node result: incomplete -> fail 2026-03-08 03:10:44,043 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:10:44,057 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69ace3f7bcd207e915d77ffc HTTP/1.1" 200 None 2026-03-08 03:10:44,059 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:10:44,072 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc01dbcd207e915d76ba7 HTTP/1.1" 200 None 2026-03-08 03:10:44,073 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:10:44,107 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69ace3f7bcd207e915d77ffc HTTP/1.1" 200 None 2026-03-08 03:10:44,198 - lava_callback - INFO - Completed processing callback for node 69ace3f7bcd207e915d77ffc 2026-03-08 03:10:45,702 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:10:45,721 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:59664 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38938 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:10:59,566 - lava_callback - INFO - Received callback for node 69aa45b18c4d487a0e752b9f INFO: 172.21.0.1:38948 - "POST /node/69aa45b18c4d487a0e752b9f HTTP/1.0" 202 Accepted 2026-03-08 03:10:59,572 - lava_callback - INFO - Starting async job submit for node 69aa45b18c4d487a0e752b9f 2026-03-08 03:10:59,573 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:10:59,600 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa45b18c4d487a0e752b9f HTTP/1.1" 200 None 2026-03-08 03:10:59,601 - lava_callback - DEBUG - Processing node 69aa45b18c4d487a0e752b9f: baseline-arm-baylibre 2026-03-08 03:10:59,601 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa45b18c4d487a0e752b9f 2026-03-08 03:10:59,601 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:10:59,622 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa45b18c4d487a0e752b9f HTTP/1.1" 200 None 2026-03-08 03:10:59,624 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:10:59,637 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6ed9c2da7ccec10683e47 HTTP/1.1" 200 None 2026-03-08 03:10:59,639 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:10:59,659 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa45b18c4d487a0e752b9f HTTP/1.1" 200 None 2026-03-08 03:10:59,661 - lava_callback - ERROR - Error processing callback for node 69aa45b18c4d487a0e752b9f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:42834 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52262 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52270 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:11:17,485 - lava_callback - INFO - Received callback for node 69acc808bcd207e915d76e94 INFO: 172.21.0.1:52284 - "POST /node/69acc808bcd207e915d76e94 HTTP/1.0" 202 Accepted 2026-03-08 03:11:17,491 - lava_callback - INFO - Starting async job submit for node 69acc808bcd207e915d76e94 2026-03-08 03:11:17,514 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:11:17,529 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc808bcd207e915d76e94 HTTP/1.1" 200 None 2026-03-08 03:11:17,530 - lava_callback - DEBUG - Processing node 69acc808bcd207e915d76e94: baseline-arm-broonie 2026-03-08 03:11:17,730 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:11:17,853 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:11:17,854 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69acc808bcd207e915d76e94/log.txt.gz 2026-03-08 03:11:17,974 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:11:18,159 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:11:18,162 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69acc808bcd207e915d76e94/lava-logs.yaml 2026-03-08 03:11:18,167 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:11:18,313 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:11:18,314 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-broonie-69acc808bcd207e915d76e94/lava_callback.json.gz 2026-03-08 03:11:18,338 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:11:18,352 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc808bcd207e915d76e94 HTTP/1.1" 200 None 2026-03-08 03:11:18,354 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:11:18,367 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 03:11:18,368 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:11:18,392 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc808bcd207e915d76e94 HTTP/1.1" 200 None 2026-03-08 03:11:18,394 - lava_callback - INFO - Completed processing callback for node 69acc808bcd207e915d76e94 INFO: 172.21.0.1:59026 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59036 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47314 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:11:39,611 - lava_callback - INFO - Received callback for node 69aa45da8c4d487a0e752bb7 INFO: 172.21.0.1:47330 - "POST /node/69aa45da8c4d487a0e752bb7 HTTP/1.0" 202 Accepted 2026-03-08 03:11:39,616 - lava_callback - INFO - Starting async job submit for node 69aa45da8c4d487a0e752bb7 2026-03-08 03:11:39,617 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:11:39,631 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa45da8c4d487a0e752bb7 HTTP/1.1" 200 None 2026-03-08 03:11:39,632 - lava_callback - DEBUG - Processing node 69aa45da8c4d487a0e752bb7: baseline-arm-baylibre 2026-03-08 03:11:39,632 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa45da8c4d487a0e752bb7 2026-03-08 03:11:39,633 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:11:39,646 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa45da8c4d487a0e752bb7 HTTP/1.1" 200 None 2026-03-08 03:11:39,648 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:11:39,662 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6ed9c2da7ccec10683e47 HTTP/1.1" 200 None 2026-03-08 03:11:39,663 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:11:39,684 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa45da8c4d487a0e752bb7 HTTP/1.1" 200 None 2026-03-08 03:11:39,690 - lava_callback - ERROR - Error processing callback for node 69aa45da8c4d487a0e752bb7: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-08 03:11:45,723 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:11:45,739 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:55844 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42252 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58614 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33020 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:33028 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55514 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55516 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47710 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55230 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56178 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43856 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36614 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:36616 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38352 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38368 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60096 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:13:39,839 - lava_callback - INFO - Received callback for node 69aa46508c4d487a0e752bf4 INFO: 172.21.0.1:38974 - "POST /node/69aa46508c4d487a0e752bf4 HTTP/1.0" 202 Accepted 2026-03-08 03:13:39,845 - lava_callback - INFO - Starting async job submit for node 69aa46508c4d487a0e752bf4 2026-03-08 03:13:39,845 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:13:39,859 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa46508c4d487a0e752bf4 HTTP/1.1" 200 None 2026-03-08 03:13:39,860 - lava_callback - DEBUG - Processing node 69aa46508c4d487a0e752bf4: baseline-arm-baylibre 2026-03-08 03:13:39,861 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa46508c4d487a0e752bf4 2026-03-08 03:13:39,861 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:13:39,874 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa46508c4d487a0e752bf4 HTTP/1.1" 200 None 2026-03-08 03:13:39,875 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:13:39,889 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6ed9c2da7ccec10683e47 HTTP/1.1" 200 None 2026-03-08 03:13:39,891 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:13:39,915 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa46508c4d487a0e752bf4 HTTP/1.1" 200 None 2026-03-08 03:13:39,917 - lava_callback - ERROR - Error processing callback for node 69aa46508c4d487a0e752bf4: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:38976 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46296 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60654 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56500 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56502 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47278 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47288 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53740 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51824 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46742 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36506 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37580 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:37588 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:15:17,700 - lava_callback - INFO - Received callback for node 69acc832bcd207e915d76f0d INFO: 172.21.0.1:37604 - "POST /node/69acc832bcd207e915d76f0d HTTP/1.0" 202 Accepted 2026-03-08 03:15:17,707 - lava_callback - INFO - Starting async job submit for node 69acc832bcd207e915d76f0d 2026-03-08 03:15:17,731 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:15:17,746 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc832bcd207e915d76f0d HTTP/1.1" 200 None 2026-03-08 03:15:17,746 - lava_callback - DEBUG - Processing node 69acc832bcd207e915d76f0d: ltp-cap_bounds 2026-03-08 03:15:18,108 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:15:18,251 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:15:18,253 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-cap_bounds-69acc832bcd207e915d76f0d/log.txt.gz 2026-03-08 03:15:18,463 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:15:18,702 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:15:18,704 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-cap_bounds-69acc832bcd207e915d76f0d/lava-logs.yaml 2026-03-08 03:15:18,713 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:15:18,876 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:15:18,878 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-cap_bounds-69acc832bcd207e915d76f0d/lava_callback.json.gz 2026-03-08 03:15:18,904 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:15:18,918 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc832bcd207e915d76f0d HTTP/1.1" 200 None 2026-03-08 03:15:18,919 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:15:18,932 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 03:15:18,934 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:15:18,968 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc832bcd207e915d76f0d HTTP/1.1" 200 None 2026-03-08 03:15:18,974 - lava_callback - INFO - Completed processing callback for node 69acc832bcd207e915d76f0d INFO: 172.21.0.1:46760 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46772 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33236 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:15:39,878 - lava_callback - INFO - Received callback for node 69aa46c78c4d487a0e752c23 INFO: 172.21.0.1:36302 - "POST /node/69aa46c78c4d487a0e752c23 HTTP/1.0" 202 Accepted 2026-03-08 03:15:39,884 - lava_callback - INFO - Starting async job submit for node 69aa46c78c4d487a0e752c23 2026-03-08 03:15:39,885 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:15:39,909 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa46c78c4d487a0e752c23 HTTP/1.1" 200 None 2026-03-08 03:15:39,910 - lava_callback - DEBUG - Processing node 69aa46c78c4d487a0e752c23: baseline-arm-baylibre 2026-03-08 03:15:39,911 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa46c78c4d487a0e752c23 2026-03-08 03:15:39,911 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:15:39,924 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa46c78c4d487a0e752c23 HTTP/1.1" 200 None 2026-03-08 03:15:39,926 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:15:39,939 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6ed9c2da7ccec10683e47 HTTP/1.1" 200 None 2026-03-08 03:15:39,940 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:15:39,961 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa46c78c4d487a0e752c23 HTTP/1.1" 200 None 2026-03-08 03:15:39,965 - lava_callback - ERROR - Error processing callback for node 69aa46c78c4d487a0e752c23: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' 2026-03-08 03:15:45,742 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:15:45,759 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:36312 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37420 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37260 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35222 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:35232 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55174 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55190 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57758 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33460 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46910 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57958 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41540 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:41554 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:17:19,955 - lava_callback - INFO - Received callback for node 69aa473c8c4d487a0e752c5f INFO: 172.21.0.1:52212 - "POST /node/69aa473c8c4d487a0e752c5f HTTP/1.0" 202 Accepted 2026-03-08 03:17:19,961 - lava_callback - INFO - Starting async job submit for node 69aa473c8c4d487a0e752c5f 2026-03-08 03:17:19,962 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:17:19,978 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa473c8c4d487a0e752c5f HTTP/1.1" 200 None 2026-03-08 03:17:19,978 - lava_callback - DEBUG - Processing node 69aa473c8c4d487a0e752c5f: baseline-arm-baylibre 2026-03-08 03:17:19,979 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa473c8c4d487a0e752c5f 2026-03-08 03:17:19,979 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:17:19,996 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa473c8c4d487a0e752c5f HTTP/1.1" 200 None 2026-03-08 03:17:19,997 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:17:20,010 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6ed9c2da7ccec10683e47 HTTP/1.1" 200 None 2026-03-08 03:17:20,012 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:17:20,032 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa473c8c4d487a0e752c5f HTTP/1.1" 200 None 2026-03-08 03:17:20,038 - lava_callback - ERROR - Error processing callback for node 69aa473c8c4d487a0e752c5f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:52224 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52240 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35384 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51350 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51640 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58076 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45656 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:45670 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46390 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46394 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44770 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56520 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44906 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49648 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42672 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:42688 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55258 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:19:22,862 - lava_callback - INFO - Received callback for node 69acc833bcd207e915d76f10 INFO: 172.21.0.1:55268 - "POST /node/69acc833bcd207e915d76f10 HTTP/1.0" 202 Accepted 2026-03-08 03:19:22,869 - lava_callback - INFO - Starting async job submit for node 69acc833bcd207e915d76f10 2026-03-08 03:19:22,893 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:19:22,908 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc833bcd207e915d76f10 HTTP/1.1" 200 None 2026-03-08 03:19:22,909 - lava_callback - DEBUG - Processing node 69acc833bcd207e915d76f10: ltp-containers 2026-03-08 03:19:23,284 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:19:23,435 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:19:23,436 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-containers-69acc833bcd207e915d76f10/log.txt.gz 2026-03-08 03:19:23,649 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:19:23,880 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:19:23,882 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-containers-69acc833bcd207e915d76f10/lava-logs.yaml 2026-03-08 03:19:23,891 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:19:24,060 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:19:24,061 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-containers-69acc833bcd207e915d76f10/lava_callback.json.gz 2026-03-08 03:19:24,087 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:19:24,114 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc833bcd207e915d76f10 HTTP/1.1" 200 None 2026-03-08 03:19:24,116 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:19:24,130 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 03:19:24,131 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:19:24,155 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc833bcd207e915d76f10 HTTP/1.1" 200 None 2026-03-08 03:19:24,160 - lava_callback - INFO - Completed processing callback for node 69acc833bcd207e915d76f10 INFO: 172.21.0.1:55278 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57122 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:19:45,762 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:19:45,778 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:40288 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43006 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56454 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59084 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59096 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56982 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56988 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38164 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52570 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48078 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41790 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46612 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46616 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46390 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46396 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44142 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44448 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58042 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56002 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41260 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:41262 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36064 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36080 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35586 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39818 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56292 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34228 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38592 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:38602 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:23:20,310 - lava_callback - INFO - Received callback for node 69aa48a68c4d487a0e752d0f INFO: 172.21.0.1:55666 - "POST /node/69aa48a68c4d487a0e752d0f HTTP/1.0" 202 Accepted 2026-03-08 03:23:20,316 - lava_callback - INFO - Starting async job submit for node 69aa48a68c4d487a0e752d0f 2026-03-08 03:23:20,317 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:23:20,341 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa48a68c4d487a0e752d0f HTTP/1.1" 200 None 2026-03-08 03:23:20,343 - lava_callback - DEBUG - Processing node 69aa48a68c4d487a0e752d0f: baseline-arm64-baylibre 2026-03-08 03:23:20,343 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa48a68c4d487a0e752d0f 2026-03-08 03:23:20,344 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:23:20,359 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa48a68c4d487a0e752d0f HTTP/1.1" 200 None 2026-03-08 03:23:20,360 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:23:20,379 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a744905a0c0e10c4819cd5 HTTP/1.1" 200 None 2026-03-08 03:23:20,381 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:23:20,412 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa48a68c4d487a0e752d0f HTTP/1.1" 200 None 2026-03-08 03:23:20,413 - lava_callback - ERROR - Error processing callback for node 69aa48a68c4d487a0e752d0f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:55670 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55676 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40558 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51548 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59054 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54470 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36186 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:36190 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51882 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51890 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41344 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57594 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37662 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39834 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44424 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44432 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38582 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38596 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33096 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46388 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43384 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49620 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46316 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46326 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40172 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40186 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53204 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60396 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34464 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:26:58,164 - lava_callback - INFO - Received callback for node 69acc834bcd207e915d76f13 INFO: 172.21.0.1:34470 - "POST /node/69acc834bcd207e915d76f13 HTTP/1.0" 202 Accepted 2026-03-08 03:26:58,171 - lava_callback - INFO - Starting async job submit for node 69acc834bcd207e915d76f13 2026-03-08 03:26:58,206 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:26:58,245 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc834bcd207e915d76f13 HTTP/1.1" 200 None 2026-03-08 03:26:58,246 - lava_callback - DEBUG - Processing node 69acc834bcd207e915d76f13: ltp-controllers 2026-03-08 03:26:58,739 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:26:58,907 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:26:58,908 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-controllers-69acc834bcd207e915d76f13/log.txt.gz 2026-03-08 03:26:59,190 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:26:59,406 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:26:59,409 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-controllers-69acc834bcd207e915d76f13/lava-logs.yaml 2026-03-08 03:26:59,420 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:26:59,579 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:26:59,581 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-controllers-69acc834bcd207e915d76f13/lava_callback.json.gz 2026-03-08 03:26:59,616 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:26:59,634 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc834bcd207e915d76f13 HTTP/1.1" 200 None 2026-03-08 03:26:59,636 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:26:59,649 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 03:26:59,651 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:26:59,679 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc834bcd207e915d76f13 HTTP/1.1" 200 None 2026-03-08 03:26:59,682 - lava_callback - INFO - Completed processing callback for node 69acc834bcd207e915d76f13 INFO: 172.21.0.1:43750 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34730 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 03:27:15,781 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:27:15,797 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:34734 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34370 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34386 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36446 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36352 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59890 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40464 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55988 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55998 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38022 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38030 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33538 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49046 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32862 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44340 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54584 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54600 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45816 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45818 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59314 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34116 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57982 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48652 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39802 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:39808 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34690 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34704 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38832 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45268 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51352 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:30:58,502 - lava_callback - INFO - Received callback for node 69acc838bcd207e915d76f1b INFO: 172.21.0.1:51356 - "POST /node/69acc838bcd207e915d76f1b HTTP/1.0" 202 Accepted 2026-03-08 03:30:58,508 - lava_callback - INFO - Starting async job submit for node 69acc838bcd207e915d76f1b 2026-03-08 03:30:58,534 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:30:58,549 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc838bcd207e915d76f1b HTTP/1.1" 200 None 2026-03-08 03:30:58,550 - lava_callback - DEBUG - Processing node 69acc838bcd207e915d76f1b: ltp-cve 2026-03-08 03:30:58,936 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:30:59,160 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:30:59,162 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-cve-69acc838bcd207e915d76f1b/log.txt.gz 2026-03-08 03:30:59,375 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:30:59,622 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:30:59,624 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-cve-69acc838bcd207e915d76f1b/lava-logs.yaml 2026-03-08 03:30:59,635 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:30:59,812 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:30:59,814 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-cve-69acc838bcd207e915d76f1b/lava_callback.json.gz 2026-03-08 03:30:59,849 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:30:59,885 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc838bcd207e915d76f1b HTTP/1.1" 200 None 2026-03-08 03:30:59,887 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:30:59,900 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 03:30:59,902 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:30:59,936 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc838bcd207e915d76f1b HTTP/1.1" 200 None 2026-03-08 03:30:59,941 - lava_callback - INFO - Completed processing callback for node 69acc838bcd207e915d76f1b INFO: 172.21.0.1:38432 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51508 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 03:31:15,800 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:31:15,824 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:51520 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49448 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49464 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39900 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33222 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36898 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52604 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53524 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:53530 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60668 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60682 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38346 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47516 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35704 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60420 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47346 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47362 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45352 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45354 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47220 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45854 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46842 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:34:00,920 - lava_callback - INFO - Received callback for node 69aa4b1c8c4d487a0e752e41 INFO: 172.21.0.1:57762 - "POST /node/69aa4b1c8c4d487a0e752e41 HTTP/1.0" 202 Accepted 2026-03-08 03:34:00,926 - lava_callback - INFO - Starting async job submit for node 69aa4b1c8c4d487a0e752e41 2026-03-08 03:34:00,927 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:34:00,940 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4b1c8c4d487a0e752e41 HTTP/1.1" 200 None 2026-03-08 03:34:00,941 - lava_callback - DEBUG - Processing node 69aa4b1c8c4d487a0e752e41: baseline-arm64-baylibre 2026-03-08 03:34:00,941 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa4b1c8c4d487a0e752e41 2026-03-08 03:34:00,942 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:34:00,954 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4b1c8c4d487a0e752e41 HTTP/1.1" 200 None 2026-03-08 03:34:00,956 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:34:00,969 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a93c777ee78ae610caa696 HTTP/1.1" 200 None 2026-03-08 03:34:00,970 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:34:00,990 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa4b1c8c4d487a0e752e41 HTTP/1.1" 200 None 2026-03-08 03:34:00,998 - lava_callback - ERROR - Error processing callback for node 69aa4b1c8c4d487a0e752e41: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:57770 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37538 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:37544 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37854 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37870 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44858 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37178 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45508 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:34:58,280 - lava_callback - INFO - Received callback for node 69acc83bbcd207e915d76f2d INFO: 172.21.0.1:45516 - "POST /node/69acc83bbcd207e915d76f2d HTTP/1.0" 202 Accepted 2026-03-08 03:34:58,287 - lava_callback - INFO - Starting async job submit for node 69acc83bbcd207e915d76f2d 2026-03-08 03:34:58,312 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:34:58,327 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc83bbcd207e915d76f2d HTTP/1.1" 200 None 2026-03-08 03:34:58,328 - lava_callback - DEBUG - Processing node 69acc83bbcd207e915d76f2d: ltp-fs_bind 2026-03-08 03:34:58,789 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:34:58,981 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:34:58,983 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-fs_bind-69acc83bbcd207e915d76f2d/log.txt.gz 2026-03-08 03:34:59,218 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:34:59,501 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:34:59,503 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-fs_bind-69acc83bbcd207e915d76f2d/lava-logs.yaml 2026-03-08 03:34:59,512 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:34:59,679 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:34:59,681 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-fs_bind-69acc83bbcd207e915d76f2d/lava_callback.json.gz 2026-03-08 03:34:59,707 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:34:59,730 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc83bbcd207e915d76f2d HTTP/1.1" 200 None 2026-03-08 03:34:59,731 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:34:59,745 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 03:34:59,747 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:34:59,775 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc83bbcd207e915d76f2d HTTP/1.1" 200 None 2026-03-08 03:34:59,779 - lava_callback - INFO - Completed processing callback for node 69acc83bbcd207e915d76f2d INFO: 172.21.0.1:40748 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44222 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 03:35:15,826 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:35:15,852 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:44236 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36450 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36460 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41454 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46730 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41518 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59952 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53090 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:53106 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36550 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36552 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47604 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33832 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41994 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:37:01,291 - lava_callback - INFO - Received callback for node 69aa4bc88c4d487a0e752e92 INFO: 172.21.0.1:39054 - "POST /node/69aa4bc88c4d487a0e752e92 HTTP/1.0" 202 Accepted 2026-03-08 03:37:01,296 - lava_callback - INFO - Starting async job submit for node 69aa4bc88c4d487a0e752e92 2026-03-08 03:37:01,297 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:37:01,311 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4bc88c4d487a0e752e92 HTTP/1.1" 200 None 2026-03-08 03:37:01,312 - lava_callback - DEBUG - Processing node 69aa4bc88c4d487a0e752e92: baseline-arm64-clabbe 2026-03-08 03:37:01,312 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa4bc88c4d487a0e752e92 2026-03-08 03:37:01,313 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:37:01,325 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4bc88c4d487a0e752e92 HTTP/1.1" 200 None 2026-03-08 03:37:01,326 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:37:01,339 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a93c777ee78ae610caa696 HTTP/1.1" 200 None 2026-03-08 03:37:01,341 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:37:01,374 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa4bc88c4d487a0e752e92 HTTP/1.1" 200 None 2026-03-08 03:37:01,376 - lava_callback - ERROR - Error processing callback for node 69aa4bc88c4d487a0e752e92: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:39064 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39438 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:39446 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49426 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49432 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38508 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37252 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56458 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49112 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51928 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:51936 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34494 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34504 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50594 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40418 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36448 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:38:58,704 - lava_callback - INFO - Received callback for node 69acc83dbcd207e915d76f34 INFO: 172.21.0.1:36464 - "POST /node/69acc83dbcd207e915d76f34 HTTP/1.0" 202 Accepted 2026-03-08 03:38:58,710 - lava_callback - INFO - Starting async job submit for node 69acc83dbcd207e915d76f34 2026-03-08 03:38:58,735 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:38:58,750 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc83dbcd207e915d76f34 HTTP/1.1" 200 None 2026-03-08 03:38:58,750 - lava_callback - DEBUG - Processing node 69acc83dbcd207e915d76f34: ltp-fsx 2026-03-08 03:38:59,175 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:38:59,331 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:38:59,332 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-fsx-69acc83dbcd207e915d76f34/log.txt.gz 2026-03-08 03:38:59,572 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:38:59,822 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:38:59,824 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-fsx-69acc83dbcd207e915d76f34/lava-logs.yaml 2026-03-08 03:38:59,833 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:39:00,014 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:39:00,016 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-fsx-69acc83dbcd207e915d76f34/lava_callback.json.gz 2026-03-08 03:39:00,041 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:39:00,064 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc83dbcd207e915d76f34 HTTP/1.1" 200 None 2026-03-08 03:39:00,066 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:39:00,081 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 03:39:00,082 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:39:00,107 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc83dbcd207e915d76f34 HTTP/1.1" 200 None 2026-03-08 03:39:00,112 - lava_callback - INFO - Completed processing callback for node 69acc83dbcd207e915d76f34 INFO: 172.21.0.1:60624 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58728 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 03:39:15,855 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:39:15,875 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:58732 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40064 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40080 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33146 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44942 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49576 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44396 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52068 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52078 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53050 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53054 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39602 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:40:41,345 - lava_callback - INFO - Received callback for node 69aa4ca78c4d487a0e752edc INFO: 172.21.0.1:39318 - "POST /node/69aa4ca78c4d487a0e752edc HTTP/1.0" 202 Accepted 2026-03-08 03:40:41,351 - lava_callback - INFO - Starting async job submit for node 69aa4ca78c4d487a0e752edc 2026-03-08 03:40:41,352 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:40:41,366 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4ca78c4d487a0e752edc HTTP/1.1" 200 None 2026-03-08 03:40:41,367 - lava_callback - DEBUG - Processing node 69aa4ca78c4d487a0e752edc: baseline-arm64-baylibre 2026-03-08 03:40:41,367 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa4ca78c4d487a0e752edc 2026-03-08 03:40:41,368 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:40:41,382 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4ca78c4d487a0e752edc HTTP/1.1" 200 None 2026-03-08 03:40:41,384 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:40:41,397 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a744905a0c0e10c4819cd5 HTTP/1.1" 200 None 2026-03-08 03:40:41,398 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:40:41,419 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa4ca78c4d487a0e752edc HTTP/1.1" 200 None 2026-03-08 03:40:41,430 - lava_callback - ERROR - Error processing callback for node 69aa4ca78c4d487a0e752edc: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:39320 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44484 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34542 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56706 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56714 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56862 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56872 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59446 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36540 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33576 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:42:01,334 - lava_callback - INFO - Received callback for node 69aa4cfc8c4d487a0e752ef9 INFO: 172.21.0.1:44888 - "POST /node/69aa4cfc8c4d487a0e752ef9 HTTP/1.0" 202 Accepted 2026-03-08 03:42:01,339 - lava_callback - INFO - Starting async job submit for node 69aa4cfc8c4d487a0e752ef9 2026-03-08 03:42:01,340 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:42:01,357 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4cfc8c4d487a0e752ef9 HTTP/1.1" 200 None 2026-03-08 03:42:01,357 - lava_callback - DEBUG - Processing node 69aa4cfc8c4d487a0e752ef9: baseline-arm64-baylibre 2026-03-08 03:42:01,358 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa4cfc8c4d487a0e752ef9 2026-03-08 03:42:01,358 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:42:01,372 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4cfc8c4d487a0e752ef9 HTTP/1.1" 200 None 2026-03-08 03:42:01,373 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:42:01,388 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6edb32da7ccec10683eaa HTTP/1.1" 200 None 2026-03-08 03:42:01,390 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:42:01,410 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa4cfc8c4d487a0e752ef9 HTTP/1.1" 200 None 2026-03-08 03:42:01,416 - lava_callback - ERROR - Error processing callback for node 69aa4cfc8c4d487a0e752ef9: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:44898 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52140 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52154 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45500 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45502 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49168 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48324 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34964 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:42:58,822 - lava_callback - INFO - Received callback for node 69acc83ebcd207e915d76f3d INFO: 172.21.0.1:34976 - "POST /node/69acc83ebcd207e915d76f3d HTTP/1.0" 202 Accepted 2026-03-08 03:42:58,828 - lava_callback - INFO - Starting async job submit for node 69acc83ebcd207e915d76f3d 2026-03-08 03:42:58,853 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:42:58,879 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc83ebcd207e915d76f3d HTTP/1.1" 200 None 2026-03-08 03:42:58,880 - lava_callback - DEBUG - Processing node 69acc83ebcd207e915d76f3d: ltp-input 2026-03-08 03:42:59,234 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:42:59,431 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:42:59,433 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-input-69acc83ebcd207e915d76f3d/log.txt.gz 2026-03-08 03:42:59,634 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:42:59,879 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:42:59,881 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-input-69acc83ebcd207e915d76f3d/lava-logs.yaml 2026-03-08 03:42:59,890 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:43:00,073 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:43:00,074 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-input-69acc83ebcd207e915d76f3d/lava_callback.json.gz 2026-03-08 03:43:00,123 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:43:00,161 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc83ebcd207e915d76f3d HTTP/1.1" 200 None 2026-03-08 03:43:00,163 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:43:00,184 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 03:43:00,186 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:43:00,213 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc83ebcd207e915d76f3d HTTP/1.1" 200 None 2026-03-08 03:43:00,214 - lava_callback - INFO - Completed processing callback for node 69acc83ebcd207e915d76f3d INFO: 172.21.0.1:43518 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48642 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 03:43:15,878 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:43:15,915 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:48656 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:43:21,572 - lava_callback - INFO - Received callback for node 69aa4d4b8c4d487a0e752f0f INFO: 172.21.0.1:42830 - "POST /node/69aa4d4b8c4d487a0e752f0f HTTP/1.0" 202 Accepted 2026-03-08 03:43:21,578 - lava_callback - INFO - Starting async job submit for node 69aa4d4b8c4d487a0e752f0f 2026-03-08 03:43:21,579 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:43:21,602 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4d4b8c4d487a0e752f0f HTTP/1.1" 200 None 2026-03-08 03:43:21,603 - lava_callback - DEBUG - Processing node 69aa4d4b8c4d487a0e752f0f: baseline-arm64-baylibre 2026-03-08 03:43:21,603 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa4d4b8c4d487a0e752f0f 2026-03-08 03:43:21,604 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:43:21,622 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4d4b8c4d487a0e752f0f HTTP/1.1" 200 None 2026-03-08 03:43:21,623 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:43:21,636 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a744905a0c0e10c4819cd5 HTTP/1.1" 200 None 2026-03-08 03:43:21,638 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:43:21,658 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa4d4b8c4d487a0e752f0f HTTP/1.1" 200 None 2026-03-08 03:43:21,663 - lava_callback - ERROR - Error processing callback for node 69aa4d4b8c4d487a0e752f0f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:42834 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42848 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46484 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50844 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43714 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34172 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59628 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:59640 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58062 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58076 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54828 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:44:41,513 - lava_callback - INFO - Received callback for node 69aa4d9d8c4d487a0e752f2c INFO: 172.21.0.1:36624 - "POST /node/69aa4d9d8c4d487a0e752f2c HTTP/1.0" 202 Accepted 2026-03-08 03:44:41,518 - lava_callback - INFO - Starting async job submit for node 69aa4d9d8c4d487a0e752f2c 2026-03-08 03:44:41,519 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:44:41,538 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4d9d8c4d487a0e752f2c HTTP/1.1" 200 None 2026-03-08 03:44:41,539 - lava_callback - DEBUG - Processing node 69aa4d9d8c4d487a0e752f2c: baseline-arm64-baylibre 2026-03-08 03:44:41,539 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa4d9d8c4d487a0e752f2c 2026-03-08 03:44:41,540 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:44:41,554 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4d9d8c4d487a0e752f2c HTTP/1.1" 200 None 2026-03-08 03:44:41,555 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:44:41,569 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a744905a0c0e10c4819cd5 HTTP/1.1" 200 None 2026-03-08 03:44:41,570 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:44:41,614 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa4d9d8c4d487a0e752f2c HTTP/1.1" 200 None 2026-03-08 03:44:41,615 - lava_callback - ERROR - Error processing callback for node 69aa4d9d8c4d487a0e752f2c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:36640 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45910 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44398 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54966 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54976 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57732 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57736 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53772 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50010 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42250 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:46:01,637 - lava_callback - INFO - Received callback for node 69aa4df28c4d487a0e752f3f INFO: 172.21.0.1:44224 - "POST /node/69aa4df28c4d487a0e752f3f HTTP/1.0" 202 Accepted 2026-03-08 03:46:01,643 - lava_callback - INFO - Starting async job submit for node 69aa4df28c4d487a0e752f3f 2026-03-08 03:46:01,644 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:46:01,658 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4df28c4d487a0e752f3f HTTP/1.1" 200 None 2026-03-08 03:46:01,659 - lava_callback - DEBUG - Processing node 69aa4df28c4d487a0e752f3f: baseline-arm64-baylibre 2026-03-08 03:46:01,659 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa4df28c4d487a0e752f3f 2026-03-08 03:46:01,660 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:46:01,674 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4df28c4d487a0e752f3f HTTP/1.1" 200 None 2026-03-08 03:46:01,675 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:46:01,688 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6edb32da7ccec10683eaa HTTP/1.1" 200 None 2026-03-08 03:46:01,690 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:46:01,715 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa4df28c4d487a0e752f3f HTTP/1.1" 200 None 2026-03-08 03:46:01,720 - lava_callback - ERROR - Error processing callback for node 69aa4df28c4d487a0e752f3f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:44228 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32776 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:32788 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60074 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60086 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47886 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33048 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58066 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:46:59,281 - lava_callback - INFO - Received callback for node 69acc841bcd207e915d76f40 INFO: 172.21.0.1:58070 - "POST /node/69acc841bcd207e915d76f40 HTTP/1.0" 202 Accepted 2026-03-08 03:46:59,288 - lava_callback - INFO - Starting async job submit for node 69acc841bcd207e915d76f40 2026-03-08 03:46:59,313 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:46:59,343 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc841bcd207e915d76f40 HTTP/1.1" 200 None 2026-03-08 03:46:59,343 - lava_callback - DEBUG - Processing node 69acc841bcd207e915d76f40: ltp-ipc 2026-03-08 03:46:59,687 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:46:59,878 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:46:59,880 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-ipc-69acc841bcd207e915d76f40/log.txt.gz 2026-03-08 03:47:00,081 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:47:00,311 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:47:00,313 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-ipc-69acc841bcd207e915d76f40/lava-logs.yaml 2026-03-08 03:47:00,322 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:47:00,501 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:47:00,503 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-ipc-69acc841bcd207e915d76f40/lava_callback.json.gz 2026-03-08 03:47:00,528 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:47:00,543 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc841bcd207e915d76f40 HTTP/1.1" 200 None 2026-03-08 03:47:00,544 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:47:00,558 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 03:47:00,559 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:47:00,596 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc841bcd207e915d76f40 HTTP/1.1" 200 None 2026-03-08 03:47:00,597 - lava_callback - INFO - Completed processing callback for node 69acc841bcd207e915d76f40 INFO: 172.21.0.1:34572 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57490 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 03:47:15,918 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:47:15,936 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:57492 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:47:21,672 - lava_callback - INFO - Received callback for node 69aa4e458c4d487a0e752f56 INFO: 172.21.0.1:53726 - "POST /node/69aa4e458c4d487a0e752f56 HTTP/1.0" 202 Accepted 2026-03-08 03:47:21,678 - lava_callback - INFO - Starting async job submit for node 69aa4e458c4d487a0e752f56 2026-03-08 03:47:21,679 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:47:21,707 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4e458c4d487a0e752f56 HTTP/1.1" 200 None 2026-03-08 03:47:21,709 - lava_callback - DEBUG - Processing node 69aa4e458c4d487a0e752f56: baseline-arm64-baylibre 2026-03-08 03:47:21,709 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa4e458c4d487a0e752f56 2026-03-08 03:47:21,710 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:47:21,745 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4e458c4d487a0e752f56 HTTP/1.1" 200 None 2026-03-08 03:47:21,747 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:47:21,763 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a744905a0c0e10c4819cd5 HTTP/1.1" 200 None 2026-03-08 03:47:21,765 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:47:21,784 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa4e458c4d487a0e752f56 HTTP/1.1" 200 None 2026-03-08 03:47:21,785 - lava_callback - ERROR - Error processing callback for node 69aa4e458c4d487a0e752f56: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:53738 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53744 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40336 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41182 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43174 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43684 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43346 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:43362 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44158 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44170 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58272 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43216 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52462 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:49:01,738 - lava_callback - INFO - Received callback for node 69aa4e998c4d487a0e752f6f INFO: 172.21.0.1:48910 - "POST /node/69aa4e998c4d487a0e752f6f HTTP/1.0" 202 Accepted 2026-03-08 03:49:01,743 - lava_callback - INFO - Starting async job submit for node 69aa4e998c4d487a0e752f6f 2026-03-08 03:49:01,744 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:49:01,761 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4e998c4d487a0e752f6f HTTP/1.1" 200 None 2026-03-08 03:49:01,762 - lava_callback - DEBUG - Processing node 69aa4e998c4d487a0e752f6f: baseline-arm64-baylibre 2026-03-08 03:49:01,762 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa4e998c4d487a0e752f6f 2026-03-08 03:49:01,763 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:49:01,782 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4e998c4d487a0e752f6f HTTP/1.1" 200 None 2026-03-08 03:49:01,783 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:49:01,815 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a744905a0c0e10c4819cd5 HTTP/1.1" 200 None 2026-03-08 03:49:01,816 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:49:01,844 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa4e998c4d487a0e752f6f HTTP/1.1" 200 None 2026-03-08 03:49:01,845 - lava_callback - ERROR - Error processing callback for node 69aa4e998c4d487a0e752f6f: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:48920 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58006 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:58012 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52054 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52062 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59912 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55942 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43228 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60206 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44326 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44342 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:50:21,740 - lava_callback - INFO - Received callback for node 69aa4eed8c4d487a0e752f90 INFO: 172.21.0.1:36296 - "POST /node/69aa4eed8c4d487a0e752f90 HTTP/1.0" 202 Accepted 2026-03-08 03:50:21,745 - lava_callback - INFO - Starting async job submit for node 69aa4eed8c4d487a0e752f90 2026-03-08 03:50:21,746 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:50:21,761 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4eed8c4d487a0e752f90 HTTP/1.1" 200 None 2026-03-08 03:50:21,761 - lava_callback - DEBUG - Processing node 69aa4eed8c4d487a0e752f90: baseline-arm64-baylibre 2026-03-08 03:50:21,762 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa4eed8c4d487a0e752f90 2026-03-08 03:50:21,762 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:50:21,778 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4eed8c4d487a0e752f90 HTTP/1.1" 200 None 2026-03-08 03:50:21,779 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:50:21,794 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a744905a0c0e10c4819cd5 HTTP/1.1" 200 None 2026-03-08 03:50:21,795 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:50:21,816 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa4eed8c4d487a0e752f90 HTTP/1.1" 200 None 2026-03-08 03:50:21,817 - lava_callback - ERROR - Error processing callback for node 69aa4eed8c4d487a0e752f90: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:36304 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36306 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44094 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33118 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46552 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:50:58,853 - lava_callback - INFO - Received callback for node 69acc843bcd207e915d76f43 INFO: 172.21.0.1:46562 - "POST /node/69acc843bcd207e915d76f43 HTTP/1.0" 202 Accepted 2026-03-08 03:50:58,863 - lava_callback - INFO - Starting async job submit for node 69acc843bcd207e915d76f43 2026-03-08 03:50:58,893 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:50:58,915 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc843bcd207e915d76f43 HTTP/1.1" 200 None 2026-03-08 03:50:58,916 - lava_callback - DEBUG - Processing node 69acc843bcd207e915d76f43: ltp-pty 2026-03-08 03:50:59,289 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:50:59,447 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:50:59,449 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-pty-69acc843bcd207e915d76f43/log.txt.gz 2026-03-08 03:50:59,651 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:50:59,898 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:50:59,901 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-pty-69acc843bcd207e915d76f43/lava-logs.yaml 2026-03-08 03:50:59,913 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:51:00,111 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:51:00,113 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-pty-69acc843bcd207e915d76f43/lava_callback.json.gz 2026-03-08 03:51:00,141 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:51:00,164 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc843bcd207e915d76f43 HTTP/1.1" 200 None 2026-03-08 03:51:00,165 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:51:00,179 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 03:51:00,181 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:51:00,208 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc843bcd207e915d76f43 HTTP/1.1" 200 None 2026-03-08 03:51:00,210 - lava_callback - INFO - Completed processing callback for node 69acc843bcd207e915d76f43 INFO: 172.21.0.1:57946 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53844 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 03:51:15,939 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:51:15,963 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:53856 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35760 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35766 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50956 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48170 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56966 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40846 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57698 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:57706 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50272 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50286 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57626 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:32826 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49892 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41448 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55544 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55558 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:53:22,014 - lava_callback - INFO - Received callback for node 69aa4fa48c4d487a0e752fe1 INFO: 172.21.0.1:56538 - "POST /node/69aa4fa48c4d487a0e752fe1 HTTP/1.0" 202 Accepted 2026-03-08 03:53:22,019 - lava_callback - INFO - Starting async job submit for node 69aa4fa48c4d487a0e752fe1 2026-03-08 03:53:22,020 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:53:22,035 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4fa48c4d487a0e752fe1 HTTP/1.1" 200 None 2026-03-08 03:53:22,036 - lava_callback - DEBUG - Processing node 69aa4fa48c4d487a0e752fe1: baseline-arm-baylibre 2026-03-08 03:53:22,036 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa4fa48c4d487a0e752fe1 2026-03-08 03:53:22,036 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:53:22,049 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4fa48c4d487a0e752fe1 HTTP/1.1" 200 None 2026-03-08 03:53:22,051 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:53:22,065 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a7447b5a0c0e10c4819cc3 HTTP/1.1" 200 None 2026-03-08 03:53:22,066 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:53:22,097 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa4fa48c4d487a0e752fe1 HTTP/1.1" 200 None 2026-03-08 03:53:22,099 - lava_callback - ERROR - Error processing callback for node 69aa4fa48c4d487a0e752fe1: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:56546 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56562 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46956 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60402 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50490 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:54:01,969 - lava_callback - INFO - Received callback for node 69aa4fd18c4d487a0e752ff6 INFO: 172.21.0.1:51052 - "POST /node/69aa4fd18c4d487a0e752ff6 HTTP/1.0" 202 Accepted 2026-03-08 03:54:01,974 - lava_callback - INFO - Starting async job submit for node 69aa4fd18c4d487a0e752ff6 2026-03-08 03:54:01,975 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:54:01,993 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4fd18c4d487a0e752ff6 HTTP/1.1" 200 None 2026-03-08 03:54:01,993 - lava_callback - DEBUG - Processing node 69aa4fd18c4d487a0e752ff6: baseline-arm-baylibre 2026-03-08 03:54:01,994 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa4fd18c4d487a0e752ff6 2026-03-08 03:54:01,994 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:54:02,010 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4fd18c4d487a0e752ff6 HTTP/1.1" 200 None 2026-03-08 03:54:02,012 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:54:02,025 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a7447b5a0c0e10c4819cc3 HTTP/1.1" 200 None 2026-03-08 03:54:02,026 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:54:02,046 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa4fd18c4d487a0e752ff6 HTTP/1.1" 200 None 2026-03-08 03:54:02,055 - lava_callback - ERROR - Error processing callback for node 69aa4fd18c4d487a0e752ff6: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:51064 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60082 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:60092 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37882 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37884 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38880 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:54:41,999 - lava_callback - INFO - Received callback for node 69aa4ffd8c4d487a0e75300c INFO: 172.21.0.1:49590 - "POST /node/69aa4ffd8c4d487a0e75300c HTTP/1.0" 202 Accepted 2026-03-08 03:54:42,006 - lava_callback - INFO - Starting async job submit for node 69aa4ffd8c4d487a0e75300c 2026-03-08 03:54:42,007 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:54:42,029 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4ffd8c4d487a0e75300c HTTP/1.1" 200 None 2026-03-08 03:54:42,030 - lava_callback - DEBUG - Processing node 69aa4ffd8c4d487a0e75300c: baseline-arm-baylibre 2026-03-08 03:54:42,030 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa4ffd8c4d487a0e75300c 2026-03-08 03:54:42,030 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:54:42,044 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa4ffd8c4d487a0e75300c HTTP/1.1" 200 None 2026-03-08 03:54:42,045 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:54:42,068 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a7447b5a0c0e10c4819cc3 HTTP/1.1" 200 None 2026-03-08 03:54:42,069 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:54:42,089 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa4ffd8c4d487a0e75300c HTTP/1.1" 200 None 2026-03-08 03:54:42,096 - lava_callback - ERROR - Error processing callback for node 69aa4ffd8c4d487a0e75300c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:49600 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47006 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:54:59,015 - lava_callback - INFO - Received callback for node 69acc844bcd207e915d76f46 INFO: 172.21.0.1:47014 - "POST /node/69acc844bcd207e915d76f46 HTTP/1.0" 202 Accepted 2026-03-08 03:54:59,021 - lava_callback - INFO - Starting async job submit for node 69acc844bcd207e915d76f46 2026-03-08 03:54:59,045 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:54:59,060 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc844bcd207e915d76f46 HTTP/1.1" 200 None 2026-03-08 03:54:59,061 - lava_callback - DEBUG - Processing node 69acc844bcd207e915d76f46: ltp-sched 2026-03-08 03:54:59,396 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:54:59,554 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:54:59,556 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-sched-69acc844bcd207e915d76f46/log.txt.gz 2026-03-08 03:54:59,761 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:54:59,994 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:54:59,996 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-sched-69acc844bcd207e915d76f46/lava-logs.yaml 2026-03-08 03:55:00,005 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:55:00,162 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:55:00,164 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-sched-69acc844bcd207e915d76f46/lava_callback.json.gz 2026-03-08 03:55:00,255 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:55:00,282 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc844bcd207e915d76f46 HTTP/1.1" 200 None 2026-03-08 03:55:00,285 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:55:00,303 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 03:55:00,304 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:55:00,338 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc844bcd207e915d76f46 HTTP/1.1" 200 None 2026-03-08 03:55:00,340 - lava_callback - INFO - Completed processing callback for node 69acc844bcd207e915d76f46 INFO: 172.21.0.1:40226 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41934 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 03:55:15,966 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:55:15,982 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:41948 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59232 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59240 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34810 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:55:42,021 - lava_callback - INFO - Received callback for node 69aa50328c4d487a0e753026 INFO: 172.21.0.1:47840 - "POST /node/69aa50328c4d487a0e753026 HTTP/1.0" 202 Accepted 2026-03-08 03:55:42,027 - lava_callback - INFO - Starting async job submit for node 69aa50328c4d487a0e753026 2026-03-08 03:55:42,028 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:55:42,042 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa50328c4d487a0e753026 HTTP/1.1" 200 None 2026-03-08 03:55:42,043 - lava_callback - DEBUG - Processing node 69aa50328c4d487a0e753026: baseline-arm-baylibre 2026-03-08 03:55:42,043 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa50328c4d487a0e753026 2026-03-08 03:55:42,044 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:55:42,056 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa50328c4d487a0e753026 HTTP/1.1" 200 None 2026-03-08 03:55:42,058 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:55:42,071 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a7447b5a0c0e10c4819cc3 HTTP/1.1" 200 None 2026-03-08 03:55:42,073 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:55:42,094 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa50328c4d487a0e753026 HTTP/1.1" 200 None 2026-03-08 03:55:42,095 - lava_callback - ERROR - Error processing callback for node 69aa50328c4d487a0e753026: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:47844 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38322 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56164 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44370 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44380 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:56:22,290 - lava_callback - INFO - Received callback for node 69aa505b8c4d487a0e753038 INFO: 172.21.0.1:42704 - "POST /node/69aa505b8c4d487a0e753038 HTTP/1.0" 202 Accepted 2026-03-08 03:56:22,295 - lava_callback - INFO - Starting async job submit for node 69aa505b8c4d487a0e753038 2026-03-08 03:56:22,296 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:56:22,318 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa505b8c4d487a0e753038 HTTP/1.1" 200 None 2026-03-08 03:56:22,319 - lava_callback - DEBUG - Processing node 69aa505b8c4d487a0e753038: baseline-arm-baylibre 2026-03-08 03:56:22,319 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa505b8c4d487a0e753038 2026-03-08 03:56:22,320 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:56:22,333 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa505b8c4d487a0e753038 HTTP/1.1" 200 None 2026-03-08 03:56:22,334 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:56:22,354 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a7447b5a0c0e10c4819cc3 HTTP/1.1" 200 None 2026-03-08 03:56:22,355 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:56:22,375 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa505b8c4d487a0e753038 HTTP/1.1" 200 None 2026-03-08 03:56:22,377 - lava_callback - ERROR - Error processing callback for node 69aa505b8c4d487a0e753038: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:42706 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42714 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38178 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51860 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45936 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:57:02,103 - lava_callback - INFO - Received callback for node 69aa50848c4d487a0e75304d INFO: 172.21.0.1:45884 - "POST /node/69aa50848c4d487a0e75304d HTTP/1.0" 202 Accepted 2026-03-08 03:57:02,108 - lava_callback - INFO - Starting async job submit for node 69aa50848c4d487a0e75304d 2026-03-08 03:57:02,109 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:57:02,124 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa50848c4d487a0e75304d HTTP/1.1" 200 None 2026-03-08 03:57:02,125 - lava_callback - DEBUG - Processing node 69aa50848c4d487a0e75304d: baseline-arm-baylibre 2026-03-08 03:57:02,125 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa50848c4d487a0e75304d 2026-03-08 03:57:02,126 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:57:02,148 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa50848c4d487a0e75304d HTTP/1.1" 200 None 2026-03-08 03:57:02,149 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:57:02,163 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a7447b5a0c0e10c4819cc3 HTTP/1.1" 200 None 2026-03-08 03:57:02,165 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:57:02,191 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa50848c4d487a0e75304d HTTP/1.1" 200 None 2026-03-08 03:57:02,192 - lava_callback - ERROR - Error processing callback for node 69aa50848c4d487a0e75304d: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:45890 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54434 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:54440 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56964 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56980 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55720 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53466 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44056 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33086 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46372 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46382 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55612 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55624 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37986 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46384 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55384 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:58:59,118 - lava_callback - INFO - Received callback for node 69acc846bcd207e915d76f49 INFO: 172.21.0.1:55400 - "POST /node/69acc846bcd207e915d76f49 HTTP/1.0" 202 Accepted 2026-03-08 03:58:59,124 - lava_callback - INFO - Starting async job submit for node 69acc846bcd207e915d76f49 2026-03-08 03:58:59,149 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:58:59,164 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc846bcd207e915d76f49 HTTP/1.1" 200 None 2026-03-08 03:58:59,165 - lava_callback - DEBUG - Processing node 69acc846bcd207e915d76f49: ltp-smoke 2026-03-08 03:58:59,525 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:58:59,673 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:58:59,675 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoke-69acc846bcd207e915d76f49/log.txt.gz 2026-03-08 03:58:59,886 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:59:00,123 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:59:00,126 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-smoke-69acc846bcd207e915d76f49/lava-logs.yaml 2026-03-08 03:59:00,135 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 03:59:00,316 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 03:59:00,318 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoke-69acc846bcd207e915d76f49/lava_callback.json.gz 2026-03-08 03:59:00,343 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:59:00,358 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc846bcd207e915d76f49 HTTP/1.1" 200 None 2026-03-08 03:59:00,359 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:59:00,379 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 03:59:00,381 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:59:00,406 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc846bcd207e915d76f49 HTTP/1.1" 200 None 2026-03-08 03:59:00,414 - lava_callback - INFO - Completed processing callback for node 69acc846bcd207e915d76f49 INFO: 172.21.0.1:55786 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43678 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 03:59:15,984 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:59:16,005 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:43688 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40066 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40078 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55636 - "GET / HTTP/1.0" 200 OK 2026-03-08 03:59:42,409 - lava_callback - INFO - Received callback for node 69aa512b8c4d487a0e75306c INFO: 172.21.0.1:44782 - "POST /node/69aa512b8c4d487a0e75306c HTTP/1.0" 202 Accepted 2026-03-08 03:59:42,414 - lava_callback - INFO - Starting async job submit for node 69aa512b8c4d487a0e75306c 2026-03-08 03:59:42,415 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:59:42,437 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa512b8c4d487a0e75306c HTTP/1.1" 200 None 2026-03-08 03:59:42,438 - lava_callback - DEBUG - Processing node 69aa512b8c4d487a0e75306c: baseline-arm64-baylibre 2026-03-08 03:59:42,438 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa512b8c4d487a0e75306c 2026-03-08 03:59:42,439 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:59:42,452 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa512b8c4d487a0e75306c HTTP/1.1" 200 None 2026-03-08 03:59:42,454 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:59:42,475 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6edb32da7ccec10683eaa HTTP/1.1" 200 None 2026-03-08 03:59:42,477 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 03:59:42,497 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa512b8c4d487a0e75306c HTTP/1.1" 200 None 2026-03-08 03:59:42,506 - lava_callback - ERROR - Error processing callback for node 69aa512b8c4d487a0e75306c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:44792 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58836 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46858 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47618 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47632 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41720 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41734 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59754 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36314 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35670 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37256 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47086 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47094 - "GET / HTTP/1.0" 200 OK 2026-03-08 04:01:22,352 - lava_callback - INFO - Received callback for node 69aa51808c4d487a0e753091 INFO: 172.21.0.1:37588 - "POST /node/69aa51808c4d487a0e753091 HTTP/1.0" 202 Accepted 2026-03-08 04:01:22,362 - lava_callback - INFO - Starting async job submit for node 69aa51808c4d487a0e753091 2026-03-08 04:01:22,363 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:01:22,390 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa51808c4d487a0e753091 HTTP/1.1" 200 None 2026-03-08 04:01:22,391 - lava_callback - DEBUG - Processing node 69aa51808c4d487a0e753091: baseline-arm64-baylibre 2026-03-08 04:01:22,392 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa51808c4d487a0e753091 2026-03-08 04:01:22,393 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:01:22,415 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa51808c4d487a0e753091 HTTP/1.1" 200 None 2026-03-08 04:01:22,417 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:01:22,448 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6edb32da7ccec10683eaa HTTP/1.1" 200 None 2026-03-08 04:01:22,450 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:01:22,488 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa51808c4d487a0e753091 HTTP/1.1" 200 None 2026-03-08 04:01:22,489 - lava_callback - ERROR - Error processing callback for node 69aa51808c4d487a0e753091: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:37594 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37604 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53652 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45958 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53074 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39050 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34452 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34456 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59786 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59788 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37438 - "GET / HTTP/1.0" 200 OK 2026-03-08 04:02:42,433 - lava_callback - INFO - Received callback for node 69aa51d38c4d487a0e7530b3 INFO: 172.21.0.1:35176 - "POST /node/69aa51d38c4d487a0e7530b3 HTTP/1.0" 202 Accepted 2026-03-08 04:02:42,438 - lava_callback - INFO - Starting async job submit for node 69aa51d38c4d487a0e7530b3 2026-03-08 04:02:42,439 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:02:42,454 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa51d38c4d487a0e7530b3 HTTP/1.1" 200 None 2026-03-08 04:02:42,455 - lava_callback - DEBUG - Processing node 69aa51d38c4d487a0e7530b3: baseline-arm64-baylibre 2026-03-08 04:02:42,455 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa51d38c4d487a0e7530b3 2026-03-08 04:02:42,455 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:02:42,468 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa51d38c4d487a0e7530b3 HTTP/1.1" 200 None 2026-03-08 04:02:42,470 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:02:42,483 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6edb32da7ccec10683eaa HTTP/1.1" 200 None 2026-03-08 04:02:42,485 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:02:42,505 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa51d38c4d487a0e7530b3 HTTP/1.1" 200 None 2026-03-08 04:02:42,509 - lava_callback - ERROR - Error processing callback for node 69aa51d38c4d487a0e7530b3: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:35180 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60192 - "GET / HTTP/1.0" 200 OK 2026-03-08 04:02:59,371 - lava_callback - INFO - Received callback for node 69acc849bcd207e915d76f4f INFO: 172.21.0.1:60208 - "POST /node/69acc849bcd207e915d76f4f HTTP/1.0" 202 Accepted 2026-03-08 04:02:59,377 - lava_callback - INFO - Starting async job submit for node 69acc849bcd207e915d76f4f 2026-03-08 04:02:59,405 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:02:59,423 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc849bcd207e915d76f4f HTTP/1.1" 200 None 2026-03-08 04:02:59,424 - lava_callback - DEBUG - Processing node 69acc849bcd207e915d76f4f: ltp-syscalls-ipc 2026-03-08 04:02:59,799 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 04:02:59,987 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 04:02:59,989 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-syscalls-ipc-69acc849bcd207e915d76f4f/log.txt.gz 2026-03-08 04:03:00,281 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 04:03:00,555 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 04:03:00,557 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-syscalls-ipc-69acc849bcd207e915d76f4f/lava-logs.yaml 2026-03-08 04:03:00,568 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 04:03:00,736 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 04:03:00,738 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-syscalls-ipc-69acc849bcd207e915d76f4f/lava_callback.json.gz 2026-03-08 04:03:00,763 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:03:00,777 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc849bcd207e915d76f4f HTTP/1.1" 200 None 2026-03-08 04:03:00,778 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:03:00,792 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 04:03:00,793 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:03:00,818 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc849bcd207e915d76f4f HTTP/1.1" 200 None 2026-03-08 04:03:00,820 - lava_callback - INFO - Completed processing callback for node 69acc849bcd207e915d76f4f INFO: 172.21.0.1:56388 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50710 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 04:03:16,008 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:03:16,025 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:50716 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57754 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57760 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51086 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48612 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52072 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55934 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43308 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:43318 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44616 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44620 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43318 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36070 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47062 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34598 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46172 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:46174 - "GET / HTTP/1.0" 200 OK 2026-03-08 04:05:22,597 - lava_callback - INFO - Received callback for node 69aa52728c4d487a0e753117 INFO: 172.21.0.1:54982 - "POST /node/69aa52728c4d487a0e753117 HTTP/1.0" 202 Accepted 2026-03-08 04:05:22,602 - lava_callback - INFO - Starting async job submit for node 69aa52728c4d487a0e753117 2026-03-08 04:05:22,603 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:05:22,617 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa52728c4d487a0e753117 HTTP/1.1" 200 None 2026-03-08 04:05:22,618 - lava_callback - DEBUG - Processing node 69aa52728c4d487a0e753117: baseline-arm64-baylibre 2026-03-08 04:05:22,618 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa52728c4d487a0e753117 2026-03-08 04:05:22,619 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:05:22,632 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa52728c4d487a0e753117 HTTP/1.1" 200 None 2026-03-08 04:05:22,634 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:05:22,647 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6edb32da7ccec10683eaa HTTP/1.1" 200 None 2026-03-08 04:05:22,649 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:05:22,669 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa52728c4d487a0e753117 HTTP/1.1" 200 None 2026-03-08 04:05:22,675 - lava_callback - ERROR - Error processing callback for node 69aa52728c4d487a0e753117: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:54998 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55008 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53308 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55282 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60756 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43764 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34782 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:34796 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47306 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47320 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53918 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58482 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47514 - "GET / HTTP/1.0" 200 OK 2026-03-08 04:06:59,460 - lava_callback - INFO - Received callback for node 69acc84abcd207e915d76f52 INFO: 172.21.0.1:47516 - "POST /node/69acc84abcd207e915d76f52 HTTP/1.0" 202 Accepted 2026-03-08 04:06:59,467 - lava_callback - INFO - Starting async job submit for node 69acc84abcd207e915d76f52 2026-03-08 04:06:59,494 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:06:59,509 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc84abcd207e915d76f52 HTTP/1.1" 200 None 2026-03-08 04:06:59,510 - lava_callback - DEBUG - Processing node 69acc84abcd207e915d76f52: ltp-timers 2026-03-08 04:06:59,853 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 04:07:00,011 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 04:07:00,013 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-timers-69acc84abcd207e915d76f52/log.txt.gz 2026-03-08 04:07:00,213 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 04:07:00,460 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 04:07:00,462 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-timers-69acc84abcd207e915d76f52/lava-logs.yaml 2026-03-08 04:07:00,471 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 04:07:00,666 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 04:07:00,668 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-timers-69acc84abcd207e915d76f52/lava_callback.json.gz 2026-03-08 04:07:00,693 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:07:00,707 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc84abcd207e915d76f52 HTTP/1.1" 200 None 2026-03-08 04:07:00,709 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:07:00,722 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc009bcd207e915d76b94 HTTP/1.1" 200 None 2026-03-08 04:07:00,723 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:07:00,748 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69acc84abcd207e915d76f52 HTTP/1.1" 200 None 2026-03-08 04:07:00,753 - lava_callback - INFO - Completed processing callback for node 69acc84abcd207e915d76f52 INFO: 172.21.0.1:48364 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58526 - "GET /api/metrics HTTP/1.0" 200 OK 2026-03-08 04:07:16,028 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:07:16,047 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "POST /latest/telemetry HTTP/1.1" 200 14 INFO: 172.21.0.1:58538 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50796 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50802 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36600 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40796 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:53932 - "GET / HTTP/1.0" 200 OK 2026-03-08 04:08:02,777 - lava_callback - INFO - Received callback for node 69aa53118c4d487a0e75315c INFO: 172.21.0.1:48998 - "POST /node/69aa53118c4d487a0e75315c HTTP/1.0" 202 Accepted 2026-03-08 04:08:02,783 - lava_callback - INFO - Starting async job submit for node 69aa53118c4d487a0e75315c 2026-03-08 04:08:02,784 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:08:02,798 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa53118c4d487a0e75315c HTTP/1.1" 200 None 2026-03-08 04:08:02,799 - lava_callback - DEBUG - Processing node 69aa53118c4d487a0e75315c: baseline-arm64-baylibre 2026-03-08 04:08:02,799 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa53118c4d487a0e75315c 2026-03-08 04:08:02,800 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:08:02,826 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa53118c4d487a0e75315c HTTP/1.1" 200 None 2026-03-08 04:08:02,829 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:08:02,842 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6edb32da7ccec10683eaa HTTP/1.1" 200 None 2026-03-08 04:08:02,844 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:08:02,879 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa53118c4d487a0e75315c HTTP/1.1" 200 None 2026-03-08 04:08:02,887 - lava_callback - ERROR - Error processing callback for node 69aa53118c4d487a0e75315c: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:49002 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49492 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:49498 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36676 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36692 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34890 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54146 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50426 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:46744 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52288 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52302 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39568 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39580 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55490 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47332 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47566 - "GET / HTTP/1.0" 200 OK 2026-03-08 04:10:02,933 - lava_callback - INFO - Received callback for node 69aa53878c4d487a0e753196 INFO: 172.21.0.1:46606 - "POST /node/69aa53878c4d487a0e753196 HTTP/1.0" 202 Accepted 2026-03-08 04:10:02,942 - lava_callback - INFO - Starting async job submit for node 69aa53878c4d487a0e753196 2026-03-08 04:10:02,943 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:10:02,966 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa53878c4d487a0e753196 HTTP/1.1" 200 None 2026-03-08 04:10:02,968 - lava_callback - DEBUG - Processing node 69aa53878c4d487a0e753196: baseline-arm64-baylibre 2026-03-08 04:10:02,969 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 69aa53878c4d487a0e753196 2026-03-08 04:10:02,971 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:10:03,000 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69aa53878c4d487a0e753196 HTTP/1.1" 200 None 2026-03-08 04:10:03,002 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:10:03,039 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69a6edb32da7ccec10683eaa HTTP/1.1" 200 None 2026-03-08 04:10:03,041 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:10:03,070 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69aa53878c4d487a0e753196 HTTP/1.1" 200 None 2026-03-08 04:10:03,080 - lava_callback - ERROR - Error processing callback for node 69aa53878c4d487a0e753196: 'lava' Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 366, in async_job_submit _emit_callback_telemetry( File "/home/kernelci/pipeline/lava_callback.py", line 387, in _emit_callback_telemetry is_infra = job_callback.is_infra_error() if job_node.get( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 119, in is_infra_error lava_yaml = self._data['results']['lava'] ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^ KeyError: 'lava' INFO: 172.21.0.1:46618 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47928 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47940 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50822 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50832 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35236 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50606 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42654 - "GET / HTTP/1.0" 200 OK 2026-03-08 04:10:59,605 - lava_callback - INFO - Received callback for node 69acc84cbcd207e915d76f5a INFO: 172.21.0.1:42662 - "POST /node/69acc84cbcd207e915d76f5a HTTP/1.0" 202 Accepted 2026-03-08 04:10:59,611 - lava_callback - INFO - Starting async job submit for node 69acc84cbcd207e915d76f5a 2026-03-08 04:10:59,636 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2026-03-08 04:10:59,659 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69acc84cbcd207e915d76f5a HTTP/1.1" 200 None 2026-03-08 04:10:59,660 - lava_callback - DEBUG - Processing node 69acc84cbcd207e915d76f5a: ltp-watchqueue 2026-03-08 04:11:00,032 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 04:11:00,207 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 04:11:00,209 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-watchqueue-69acc84cbcd207e915d76f5a/log.txt.gz 2026-03-08 04:11:00,460 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2026-03-08 04:11:00,753 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2026-03-08 04:11:00,755 - lava_callback - INFO - LAVA YAML uploaded to https://files-staging.kernelci.org/ltp-watchqueue-69acc84cbcd207e915d76f5a/lava-logs.yaml 2026-03-08 04:11:00,763 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443