INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) INFO: 172.21.0.1:60490 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60506 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43242 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34288 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52378 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:52388 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59724 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33504 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58590 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58602 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59082 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37006 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:48684 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:48688 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47930 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57592 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39934 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39948 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44100 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:49:05,580 - lava_callback - INFO - Received callback for node 691873cabe9adb01fa8e4345 2025-11-16 21:49:05,594 - lava_callback - INFO - Starting async job submit for node 691873cabe9adb01fa8e4345 INFO: 172.21.0.1:49928 - "POST /node/691873cabe9adb01fa8e4345 HTTP/1.0" 202 Accepted 2025-11-16 21:49:06,013 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:49:06,041 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691873cabe9adb01fa8e4345 HTTP/1.1" 200 None 2025-11-16 21:49:06,041 - lava_callback - DEBUG - Processing node 691873cabe9adb01fa8e4345: ltp-controllers INFO: 172.21.0.1:49942 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:49:09,177 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:56346 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:56356 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41536 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33936 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:49:40,836 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:49:40,837 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-controllers-691873cabe9adb01fa8e4345/log.txt.gz 2025-11-16 21:49:40,901 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:49:41,238 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:49:41,239 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-controllers-691873cabe9adb01fa8e4345/lava_callback.json.gz DEBUG: 691873cabe9adb01fa8e4345 Transitting job node result: pass -> fail 2025-11-16 21:49:41,314 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:49:41,334 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691873cabe9adb01fa8e4345 HTTP/1.1" 200 None 2025-11-16 21:49:41,335 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:49:41,351 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69181a354a7acaa232fb6859 HTTP/1.1" 200 None 2025-11-16 21:49:41,356 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:49:42,205 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/691873cabe9adb01fa8e4345 HTTP/1.1" 200 None 2025-11-16 21:49:42,213 - lava_callback - INFO - Completed processing callback for node 691873cabe9adb01fa8e4345 INFO: 172.21.0.1:55716 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55722 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36024 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38218 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:47776 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47778 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57442 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43246 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:50:45,253 - lava_callback - INFO - Received callback for node 691873dcbe9adb01fa8e4351 INFO: 172.21.0.1:44998 - "POST /node/691873dcbe9adb01fa8e4351 HTTP/1.0" 202 Accepted 2025-11-16 21:50:45,261 - lava_callback - INFO - Starting async job submit for node 691873dcbe9adb01fa8e4351 2025-11-16 21:50:45,431 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:50:45,447 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691873dcbe9adb01fa8e4351 HTTP/1.1" 200 None 2025-11-16 21:50:45,447 - lava_callback - DEBUG - Processing node 691873dcbe9adb01fa8e4351: ltp-fs-bind 2025-11-16 21:50:46,125 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:50:46,289 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:50:46,290 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-fs-bind-691873dcbe9adb01fa8e4351/log.txt.gz 2025-11-16 21:50:46,305 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:50:46,471 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:50:46,472 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-fs-bind-691873dcbe9adb01fa8e4351/lava_callback.json.gz INFO: 172.21.0.1:45010 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:50:46,528 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:50:46,542 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691873dcbe9adb01fa8e4351 HTTP/1.1" 200 None 2025-11-16 21:50:46,543 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:50:46,554 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69181a354a7acaa232fb6859 HTTP/1.1" 200 None 2025-11-16 21:50:46,556 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:50:46,775 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/691873dcbe9adb01fa8e4351 HTTP/1.1" 200 None 2025-11-16 21:50:46,779 - lava_callback - INFO - Completed processing callback for node 691873dcbe9adb01fa8e4351 2025-11-16 21:50:46,863 - lava_callback - INFO - Received callback for node 691873debe9adb01fa8e4354 INFO: 172.21.0.1:45020 - "POST /node/691873debe9adb01fa8e4354 HTTP/1.0" 202 Accepted 2025-11-16 21:50:46,873 - lava_callback - INFO - Starting async job submit for node 691873debe9adb01fa8e4354 2025-11-16 21:50:46,907 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:50:46,919 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691873debe9adb01fa8e4354 HTTP/1.1" 200 None 2025-11-16 21:50:46,920 - lava_callback - DEBUG - Processing node 691873debe9adb01fa8e4354: ltp-fsx 2025-11-16 21:50:47,358 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:50:47,469 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:50:47,470 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-fsx-691873debe9adb01fa8e4354/log.txt.gz 2025-11-16 21:50:47,480 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:50:47,652 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:50:47,652 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-fsx-691873debe9adb01fa8e4354/lava_callback.json.gz 2025-11-16 21:50:47,681 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:50:47,703 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691873debe9adb01fa8e4354 HTTP/1.1" 200 None 2025-11-16 21:50:47,704 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:50:47,715 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69181a354a7acaa232fb6859 HTTP/1.1" 200 None 2025-11-16 21:50:47,717 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:50:47,747 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/691873debe9adb01fa8e4354 HTTP/1.1" 200 None 2025-11-16 21:50:47,748 - lava_callback - INFO - Completed processing callback for node 691873debe9adb01fa8e4354 INFO: 172.21.0.1:45026 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44138 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:37066 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:51:07,293 - lava_callback - INFO - Received callback for node 6917a7c2364b1cb06de83ae2 INFO: 172.21.0.1:37068 - "POST /node/6917a7c2364b1cb06de83ae2 HTTP/1.0" 202 Accepted 2025-11-16 21:51:07,301 - lava_callback - INFO - Starting async job submit for node 6917a7c2364b1cb06de83ae2 2025-11-16 21:51:07,353 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:51:07,366 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6917a7c2364b1cb06de83ae2 HTTP/1.1" 200 None 2025-11-16 21:51:07,367 - lava_callback - DEBUG - Processing node 6917a7c2364b1cb06de83ae2: baseline-arm64 2025-11-16 21:51:08,482 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:51:08,667 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:51:08,668 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-6917a7c2364b1cb06de83ae2/log.txt.gz 2025-11-16 21:51:08,691 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:51:08,798 - lava_callback - INFO - Received callback for node 6917a7c2364b1cb06de83ae8 2025-11-16 21:51:08,809 - lava_callback - INFO - Starting async job submit for node 6917a7c2364b1cb06de83ae8 INFO: 172.21.0.1:37082 - "POST /node/6917a7c2364b1cb06de83ae8 HTTP/1.0" 202 Accepted 2025-11-16 21:51:08,869 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:51:08,870 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:51:08,870 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-6917a7c2364b1cb06de83ae2/lava_callback.json.gz 2025-11-16 21:51:08,924 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6917a7c2364b1cb06de83ae8 HTTP/1.1" 200 None 2025-11-16 21:51:08,930 - lava_callback - DEBUG - Processing node 6917a7c2364b1cb06de83ae8: baseline-arm64 DEBUG: Setup failed for 6917a7c2364b1cb06de83ae2. Transitting job node result: incomplete -> fail 2025-11-16 21:51:09,093 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:51:09,146 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6917a7c2364b1cb06de83ae2 HTTP/1.1" 200 None 2025-11-16 21:51:09,147 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:51:09,213 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69120d8de314d938f23977c5 HTTP/1.1" 200 None 2025-11-16 21:51:09,220 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:51:09,312 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/6917a7c2364b1cb06de83ae2 HTTP/1.1" 200 None 2025-11-16 21:51:09,312 - lava_callback - INFO - Completed processing callback for node 6917a7c2364b1cb06de83ae2 2025-11-16 21:51:10,205 - lava_callback - INFO - Received callback for node 6917a7fc364b1cb06de83af4 INFO: 172.21.0.1:47934 - "POST /node/6917a7fc364b1cb06de83af4 HTTP/1.0" 202 Accepted 2025-11-16 21:51:10,230 - lava_callback - INFO - Starting async job submit for node 6917a7fc364b1cb06de83af4 2025-11-16 21:51:10,351 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:51:10,424 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6917a7fc364b1cb06de83af4 HTTP/1.1" 200 None 2025-11-16 21:51:10,424 - lava_callback - DEBUG - Processing node 6917a7fc364b1cb06de83af4: baseline-arm64 2025-11-16 21:51:10,490 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:51:10,818 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:51:10,818 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-6917a7c2364b1cb06de83ae8/log.txt.gz 2025-11-16 21:51:10,918 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:51:11,283 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:51:11,284 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-6917a7c2364b1cb06de83ae8/lava_callback.json.gz DEBUG: Setup failed for 6917a7c2364b1cb06de83ae8. Transitting job node result: incomplete -> fail 2025-11-16 21:51:11,614 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:51:11,677 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6917a7c2364b1cb06de83ae8 HTTP/1.1" 200 None 2025-11-16 21:51:11,678 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:51:11,729 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69120d8de314d938f23977c5 HTTP/1.1" 200 None 2025-11-16 21:51:11,733 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:51:11,816 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/6917a7c2364b1cb06de83ae8 HTTP/1.1" 200 None 2025-11-16 21:51:11,816 - lava_callback - INFO - Completed processing callback for node 6917a7c2364b1cb06de83ae8 2025-11-16 21:51:12,421 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:51:12,756 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:51:12,757 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-6917a7fc364b1cb06de83af4/log.txt.gz 2025-11-16 21:51:12,783 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:51:12,988 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:51:12,988 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-6917a7fc364b1cb06de83af4/lava_callback.json.gz DEBUG: Setup failed for 6917a7fc364b1cb06de83af4. Transitting job node result: incomplete -> fail 2025-11-16 21:51:13,130 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:51:13,144 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6917a7fc364b1cb06de83af4 HTTP/1.1" 200 None 2025-11-16 21:51:13,145 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:51:13,158 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69120d8de314d938f23977c5 HTTP/1.1" 200 None 2025-11-16 21:51:13,159 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:51:13,190 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/6917a7fc364b1cb06de83af4 HTTP/1.1" 200 None 2025-11-16 21:51:13,191 - lava_callback - INFO - Completed processing callback for node 6917a7fc364b1cb06de83af4 INFO: 172.21.0.1:47946 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:47954 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33488 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:51:29,048 - lava_callback - INFO - Received callback for node 6917a4a6364b1cb06de839fd INFO: 172.21.0.1:33502 - "POST /node/6917a4a6364b1cb06de839fd HTTP/1.0" 202 Accepted 2025-11-16 21:51:29,056 - lava_callback - INFO - Starting async job submit for node 6917a4a6364b1cb06de839fd 2025-11-16 21:51:29,098 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:51:29,111 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6917a4a6364b1cb06de839fd HTTP/1.1" 200 None 2025-11-16 21:51:29,111 - lava_callback - DEBUG - Processing node 6917a4a6364b1cb06de839fd: baseline-arm64 2025-11-16 21:51:30,228 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:51:30,396 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:51:30,396 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-6917a4a6364b1cb06de839fd/log.txt.gz 2025-11-16 21:51:30,418 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:51:30,601 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:51:30,602 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-6917a4a6364b1cb06de839fd/lava_callback.json.gz DEBUG: Setup failed for 6917a4a6364b1cb06de839fd. Transitting job node result: incomplete -> fail 2025-11-16 21:51:30,687 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:51:30,699 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6917a4a6364b1cb06de839fd HTTP/1.1" 200 None 2025-11-16 21:51:30,700 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:51:30,713 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69120d8de314d938f23977c5 HTTP/1.1" 200 None 2025-11-16 21:51:30,714 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:51:30,743 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/6917a4a6364b1cb06de839fd HTTP/1.1" 200 None 2025-11-16 21:51:30,744 - lava_callback - INFO - Completed processing callback for node 6917a4a6364b1cb06de839fd INFO: 172.21.0.1:49620 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56666 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:56676 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42448 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50290 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33226 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:33236 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60546 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52292 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34938 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34940 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58886 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:53:05,326 - lava_callback - INFO - Received callback for node 691873e0be9adb01fa8e4357 INFO: 172.21.0.1:43642 - "POST /node/691873e0be9adb01fa8e4357 HTTP/1.0" 202 Accepted 2025-11-16 21:53:05,333 - lava_callback - INFO - Starting async job submit for node 691873e0be9adb01fa8e4357 2025-11-16 21:53:05,378 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:53:05,405 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691873e0be9adb01fa8e4357 HTTP/1.1" 200 None 2025-11-16 21:53:05,405 - lava_callback - DEBUG - Processing node 691873e0be9adb01fa8e4357: ltp-input 2025-11-16 21:53:05,823 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:53:05,857 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 502 150 Upload attempt 1 failed with 502 Bad Gateway: 502 Server Error: Bad Gateway for url: https://files-staging.kernelci.org/upload Retrying in 10 seconds... (4 retries remaining) INFO: 172.21.0.1:43648 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44870 - "GET /api/metrics HTTP/1.0" 200 OK 2025-11-16 21:53:15,859 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:53:15,874 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 502 150 Upload attempt 2 failed with 502 Bad Gateway: 502 Server Error: Bad Gateway for url: https://files-staging.kernelci.org/upload Retrying in 10 seconds... (3 retries remaining) INFO: 172.21.0.1:44884 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:53:25,875 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:53:25,886 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 502 150 Upload attempt 3 failed with 502 Bad Gateway: 502 Server Error: Bad Gateway for url: https://files-staging.kernelci.org/upload Retrying in 10 seconds... (2 retries remaining) INFO: 172.21.0.1:50494 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:53:35,887 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:53:35,900 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 502 150 Upload attempt 4 failed with 502 Bad Gateway: 502 Server Error: Bad Gateway for url: https://files-staging.kernelci.org/upload Retrying in 10 seconds... (1 retries remaining) INFO: 172.21.0.1:34416 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:53:45,571 - lava_callback - INFO - Received callback for node 69186d56be9adb01fa8e3a92 INFO: 172.21.0.1:54818 - "POST /node/69186d56be9adb01fa8e3a92 HTTP/1.0" 202 Accepted 2025-11-16 21:53:45,611 - lava_callback - INFO - Starting async job submit for node 69186d56be9adb01fa8e3a92 2025-11-16 21:53:45,646 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:53:45,664 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69186d56be9adb01fa8e3a92 HTTP/1.1" 200 None 2025-11-16 21:53:45,665 - lava_callback - DEBUG - Processing node 69186d56be9adb01fa8e3a92: ltp-syscalls-ipc 2025-11-16 21:53:45,932 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:53:46,011 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 502 150 Upload failed after 5 attempts 2025-11-16 21:53:46,048 - lava_callback - ERROR - Error processing callback for node 691873e0be9adb01fa8e4357: 502 Server Error: Bad Gateway for url: https://files-staging.kernelci.org/upload Traceback (most recent call last): File "/home/kernelci/pipeline/lava_callback.py", line 283, in async_job_submit log_txt_url = _upload_log(log_parser, job_node, storage) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/lava_callback.py", line 216, in _upload_log return _upload_file(storage, job_node, src, 'log.txt.gz') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kernelci/pipeline/lava_callback.py", line 182, in _upload_file return storage.upload_single((source_name, destination_name), upload_dir) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/storage/__init__.py", line 69, in upload_single urls = self._upload([file_path], dest_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/storage/backend.py", line 118, in _upload raise last_exception File "/usr/local/lib/python3.12/site-packages/kernelci/storage/backend.py", line 84, in _upload resp.raise_for_status() File "/usr/local/lib/python3.12/site-packages/requests/models.py", line 1026, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://files-staging.kernelci.org/upload INFO: 172.21.0.1:54826 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:53:47,165 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:53:47,180 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 502 150 Upload attempt 1 failed with 502 Bad Gateway: 502 Server Error: Bad Gateway for url: https://files-staging.kernelci.org/upload Retrying in 10 seconds... (4 retries remaining) INFO: 172.21.0.1:54836 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39804 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:53:57,182 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:47660 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:54:10,948 - lava_callback - INFO - Received callback for node 6917ae86364b1cb06de84395 INFO: 172.21.0.1:44078 - "POST /node/6917ae86364b1cb06de84395 HTTP/1.0" 202 Accepted 2025-11-16 21:54:10,996 - lava_callback - INFO - Starting async job submit for node 6917ae86364b1cb06de84395 2025-11-16 21:54:11,020 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:54:11,032 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6917ae86364b1cb06de84395 HTTP/1.1" 200 None 2025-11-16 21:54:11,033 - lava_callback - DEBUG - Processing node 6917ae86364b1cb06de84395: baseline-arm-pengutronix 2025-11-16 21:54:11,142 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:54:11,155 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 502 150 Upload attempt 1 failed with 502 Bad Gateway: 502 Server Error: Bad Gateway for url: https://files-staging.kernelci.org/upload Retrying in 10 seconds... (4 retries remaining) INFO: 172.21.0.1:44084 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44088 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:54:21,157 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:54:21,473 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:54:21,474 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-syscalls-ipc-69186d56be9adb01fa8e3a92/log.txt.gz 2025-11-16 21:54:21,514 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:54:22,040 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:54:22,041 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-6917ae86364b1cb06de84395/log.txt.gz 2025-11-16 21:54:22,045 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:54:22,393 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:54:22,394 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-syscalls-ipc-69186d56be9adb01fa8e3a92/lava_callback.json.gz 2025-11-16 21:54:22,532 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:54:22,559 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69186d56be9adb01fa8e3a92 HTTP/1.1" 200 None 2025-11-16 21:54:22,560 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:54:22,561 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:54:22,561 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-pengutronix-6917ae86364b1cb06de84395/lava_callback.json.gz 2025-11-16 21:54:22,612 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6916763decccb2b3698651a9 HTTP/1.1" 200 None 2025-11-16 21:54:22,613 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 DEBUG: Setup failed for 6917ae86364b1cb06de84395. Transitting job node result: incomplete -> fail 2025-11-16 21:54:22,651 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:54:22,698 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6917ae86364b1cb06de84395 HTTP/1.1" 200 None 2025-11-16 21:54:22,700 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:54:22,719 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/69186d56be9adb01fa8e3a92 HTTP/1.1" 200 None 2025-11-16 21:54:22,720 - lava_callback - INFO - Completed processing callback for node 69186d56be9adb01fa8e3a92 2025-11-16 21:54:22,771 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69120d87e314d938f23977bb HTTP/1.1" 200 None 2025-11-16 21:54:22,772 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:54:22,850 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/6917ae86364b1cb06de84395 HTTP/1.1" 200 None 2025-11-16 21:54:22,850 - lava_callback - INFO - Completed processing callback for node 6917ae86364b1cb06de84395 INFO: 172.21.0.1:43676 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33232 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:54:43,849 - lava_callback - INFO - Received callback for node 6917a4aa364b1cb06de83a00 INFO: 172.21.0.1:55954 - "POST /node/6917a4aa364b1cb06de83a00 HTTP/1.0" 202 Accepted 2025-11-16 21:54:43,855 - lava_callback - INFO - Starting async job submit for node 6917a4aa364b1cb06de83a00 2025-11-16 21:54:43,913 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:54:43,926 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6917a4aa364b1cb06de83a00 HTTP/1.1" 200 None 2025-11-16 21:54:43,927 - lava_callback - DEBUG - Processing node 6917a4aa364b1cb06de83a00: baseline-arm64 2025-11-16 21:54:44,506 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:54:44,621 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:54:44,622 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm64-6917a4aa364b1cb06de83a00/log.txt.gz 2025-11-16 21:54:44,633 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:54:44,763 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:54:44,764 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm64-6917a4aa364b1cb06de83a00/lava_callback.json.gz DEBUG: Setup failed for 6917a4aa364b1cb06de83a00. Transitting job node result: incomplete -> fail 2025-11-16 21:54:44,834 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:54:44,848 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6917a4aa364b1cb06de83a00 HTTP/1.1" 200 None 2025-11-16 21:54:44,849 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:54:44,862 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69120d8de314d938f23977c5 HTTP/1.1" 200 None 2025-11-16 21:54:44,863 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:54:44,890 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/6917a4aa364b1cb06de83a00 HTTP/1.1" 200 None 2025-11-16 21:54:44,891 - lava_callback - INFO - Completed processing callback for node 6917a4aa364b1cb06de83a00 INFO: 172.21.0.1:55968 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55978 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35780 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38264 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51040 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:51052 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51598 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:50796 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54504 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54518 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:52796 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41956 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42412 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:42426 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:42200 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:43738 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33522 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33538 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:38612 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:57:01,698 - lava_callback - INFO - Received callback for node 691a481faecba116424ed944 INFO: 172.21.0.1:34880 - "POST /node/691a481faecba116424ed944 HTTP/1.0" 202 Accepted 2025-11-16 21:57:01,705 - lava_callback - INFO - Starting async job submit for node 691a481faecba116424ed944 2025-11-16 21:57:01,736 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:57:01,769 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691a481faecba116424ed944 HTTP/1.1" 200 None 2025-11-16 21:57:01,769 - lava_callback - DEBUG - Processing node 691a481faecba116424ed944: baseline-arm 2025-11-16 21:57:01,981 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:34884 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:57:06,684 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:57:06,685 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/baseline-arm-691a481faecba116424ed944/log.txt.gz 2025-11-16 21:57:06,691 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:57:07,029 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:57:07,029 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/baseline-arm-691a481faecba116424ed944/lava_callback.json.gz 2025-11-16 21:57:07,058 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:57:07,071 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691a481faecba116424ed944 HTTP/1.1" 200 None 2025-11-16 21:57:07,073 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:57:07,086 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691a3605df7c70114a623761 HTTP/1.1" 200 None 2025-11-16 21:57:07,088 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:57:07,121 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/691a481faecba116424ed944 HTTP/1.1" 200 None 2025-11-16 21:57:07,124 - lava_callback - INFO - Completed processing callback for node 691a481faecba116424ed944 INFO: 172.21.0.1:55698 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:55704 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57268 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:58824 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54736 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54740 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60344 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45922 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:45244 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:45246 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:58:25,532 - lava_callback - INFO - Received callback for node 691873e7be9adb01fa8e4363 INFO: 172.21.0.1:38670 - "POST /node/691873e7be9adb01fa8e4363 HTTP/1.0" 202 Accepted 2025-11-16 21:58:25,539 - lava_callback - INFO - Starting async job submit for node 691873e7be9adb01fa8e4363 2025-11-16 21:58:25,573 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:58:25,590 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691873e7be9adb01fa8e4363 HTTP/1.1" 200 None 2025-11-16 21:58:25,590 - lava_callback - DEBUG - Processing node 691873e7be9adb01fa8e4363: ltp-smoketest 2025-11-16 21:58:26,006 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:38684 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:60896 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34998 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:35002 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36822 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49114 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:59:08,720 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:59:08,721 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-smoketest-691873e7be9adb01fa8e4363/log.txt.gz 2025-11-16 21:59:08,734 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:59:09,144 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:59:09,146 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-smoketest-691873e7be9adb01fa8e4363/lava_callback.json.gz DEBUG: 691873e7be9adb01fa8e4363 Transitting job node result: pass -> fail 2025-11-16 21:59:09,211 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:59:09,234 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691873e7be9adb01fa8e4363 HTTP/1.1" 200 None 2025-11-16 21:59:09,235 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:59:09,252 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69181a354a7acaa232fb6859 HTTP/1.1" 200 None 2025-11-16 21:59:09,254 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:59:09,338 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/691873e7be9adb01fa8e4363 HTTP/1.1" 200 None 2025-11-16 21:59:09,339 - lava_callback - INFO - Completed processing callback for node 691873e7be9adb01fa8e4363 INFO: 172.21.0.1:45200 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:45216 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:57626 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33844 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:59:45,518 - lava_callback - INFO - Received callback for node 691873e5be9adb01fa8e4360 INFO: 172.21.0.1:43862 - "POST /node/691873e5be9adb01fa8e4360 HTTP/1.0" 202 Accepted 2025-11-16 21:59:45,528 - lava_callback - INFO - Starting async job submit for node 691873e5be9adb01fa8e4360 2025-11-16 21:59:45,576 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:59:45,606 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691873e5be9adb01fa8e4360 HTTP/1.1" 200 None 2025-11-16 21:59:45,607 - lava_callback - DEBUG - Processing node 691873e5be9adb01fa8e4360: ltp-sched 2025-11-16 21:59:46,183 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:59:46,322 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:59:46,323 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-sched-691873e5be9adb01fa8e4360/log.txt.gz 2025-11-16 21:59:46,336 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 21:59:46,494 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 21:59:46,495 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-sched-691873e5be9adb01fa8e4360/lava_callback.json.gz DEBUG: 691873e5be9adb01fa8e4360 Transitting job node result: pass -> fail 2025-11-16 21:59:46,549 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 INFO: 172.21.0.1:43876 - "GET / HTTP/1.0" 200 OK 2025-11-16 21:59:46,582 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691873e5be9adb01fa8e4360 HTTP/1.1" 200 None 2025-11-16 21:59:46,583 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:59:46,611 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69181a354a7acaa232fb6859 HTTP/1.1" 200 None 2025-11-16 21:59:46,613 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 21:59:46,666 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/691873e5be9adb01fa8e4360 HTTP/1.1" 200 None 2025-11-16 21:59:46,667 - lava_callback - INFO - Completed processing callback for node 691873e5be9adb01fa8e4360 INFO: 172.21.0.1:43878 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:34350 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54324 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44036 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:44042 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:44518 - "GET / HTTP/1.0" 200 OK 2025-11-16 22:00:36,086 - lava_callback - INFO - Received callback for node 6917a675364b1cb06de83a3d INFO: 172.21.0.1:47198 - "POST /node/6917a675364b1cb06de83a3d HTTP/1.0" 202 Accepted 2025-11-16 22:00:36,092 - lava_callback - INFO - Starting async job submit for node 6917a675364b1cb06de83a3d 2025-11-16 22:00:36,093 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 22:00:36,106 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6917a675364b1cb06de83a3d HTTP/1.1" 200 None 2025-11-16 22:00:36,106 - lava_callback - DEBUG - Processing node 6917a675364b1cb06de83a3d: baseline-arm64 2025-11-16 22:00:36,106 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 6917a675364b1cb06de83a3d 2025-11-16 22:00:36,107 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 22:00:36,118 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6917a675364b1cb06de83a3d HTTP/1.1" 200 None 2025-11-16 22:00:36,119 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 22:00:36,131 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69120d8de314d938f23977c5 HTTP/1.1" 200 None 2025-11-16 22:00:36,132 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 22:00:36,157 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/6917a675364b1cb06de83a3d HTTP/1.1" 200 None 2025-11-16 22:00:36,158 - lava_callback - INFO - Completed processing callback for node 6917a675364b1cb06de83a3d INFO: 172.21.0.1:47204 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54460 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:54474 - "GET / HTTP/1.0" 200 OK 2025-11-16 22:00:56,173 - lava_callback - INFO - Received callback for node 6917a680364b1cb06de83a40 INFO: 172.21.0.1:47330 - "POST /node/6917a680364b1cb06de83a40 HTTP/1.0" 202 Accepted 2025-11-16 22:00:56,182 - lava_callback - INFO - Starting async job submit for node 6917a680364b1cb06de83a40 2025-11-16 22:00:56,183 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 22:00:56,197 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6917a680364b1cb06de83a40 HTTP/1.1" 200 None 2025-11-16 22:00:56,198 - lava_callback - DEBUG - Processing node 6917a680364b1cb06de83a40: baseline-arm64 2025-11-16 22:00:56,198 - lava_callback - WARNING - No log data found in callback Job failure metadata not found for node: 6917a680364b1cb06de83a40 2025-11-16 22:00:56,198 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 22:00:56,211 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/6917a680364b1cb06de83a40 HTTP/1.1" 200 None 2025-11-16 22:00:56,212 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 22:00:56,227 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69120d8de314d938f23977c5 HTTP/1.1" 200 None 2025-11-16 22:00:56,228 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 22:00:56,407 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/6917a680364b1cb06de83a40 HTTP/1.1" 200 None 2025-11-16 22:00:56,408 - lava_callback - INFO - Completed processing callback for node 6917a680364b1cb06de83a40 INFO: 172.21.0.1:47334 - "GET / HTTP/1.0" 200 OK 2025-11-16 22:01:02,287 - lava_callback - INFO - Received callback for node 691873dabe9adb01fa8e434e INFO: 172.21.0.1:49550 - "POST /node/691873dabe9adb01fa8e434e HTTP/1.0" 202 Accepted 2025-11-16 22:01:02,295 - lava_callback - INFO - Starting async job submit for node 691873dabe9adb01fa8e434e 2025-11-16 22:01:02,411 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 22:01:02,425 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691873dabe9adb01fa8e434e HTTP/1.1" 200 None 2025-11-16 22:01:02,425 - lava_callback - DEBUG - Processing node 691873dabe9adb01fa8e434e: ltp-cve 2025-11-16 22:01:03,188 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:49564 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:33778 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:33780 - "GET / HTTP/1.0" 200 OK 2025-11-16 22:01:21,847 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 22:01:21,848 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-cve-691873dabe9adb01fa8e434e/log.txt.gz 2025-11-16 22:01:21,873 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 22:01:22,106 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 22:01:22,107 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-cve-691873dabe9adb01fa8e434e/lava_callback.json.gz DEBUG: 691873dabe9adb01fa8e434e Transitting job node result: pass -> fail 2025-11-16 22:01:22,190 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 22:01:22,212 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691873dabe9adb01fa8e434e HTTP/1.1" 200 None 2025-11-16 22:01:22,214 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 22:01:22,230 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69181a354a7acaa232fb6859 HTTP/1.1" 200 None 2025-11-16 22:01:22,232 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 22:01:22,528 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/691873dabe9adb01fa8e434e HTTP/1.1" 200 None 2025-11-16 22:01:22,534 - lava_callback - INFO - Completed processing callback for node 691873dabe9adb01fa8e434e INFO: 172.21.0.1:48576 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55700 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39572 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:39580 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:55740 - "GET / HTTP/1.0" 200 OK 2025-11-16 22:02:05,684 - lava_callback - INFO - Received callback for node 691873e2be9adb01fa8e435a INFO: 172.21.0.1:49696 - "POST /node/691873e2be9adb01fa8e435a HTTP/1.0" 202 Accepted 2025-11-16 22:02:05,691 - lava_callback - INFO - Starting async job submit for node 691873e2be9adb01fa8e435a 2025-11-16 22:02:05,747 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 22:02:05,783 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691873e2be9adb01fa8e435a HTTP/1.1" 200 None 2025-11-16 22:02:05,784 - lava_callback - DEBUG - Processing node 691873e2be9adb01fa8e435a: ltp-ipc 2025-11-16 22:02:06,452 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 22:02:06,464 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 502 150 Upload attempt 1 failed with 502 Bad Gateway: 502 Server Error: Bad Gateway for url: https://files-staging.kernelci.org/upload Retrying in 10 seconds... (4 retries remaining) INFO: 172.21.0.1:49700 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:51452 - "GET /api/metrics HTTP/1.0" 200 OK 2025-11-16 22:02:16,465 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 22:02:16,499 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 502 150 Upload attempt 2 failed with 502 Bad Gateway: 502 Server Error: Bad Gateway for url: https://files-staging.kernelci.org/upload Retrying in 10 seconds... (3 retries remaining) INFO: 172.21.0.1:51464 - "GET / HTTP/1.0" 200 OK 2025-11-16 22:02:26,501 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:34096 - "GET / HTTP/1.0" 200 OK 2025-11-16 22:02:26,667 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 22:02:26,668 - lava_callback - INFO - Log uploaded to https://files-staging.kernelci.org/ltp-ipc-691873e2be9adb01fa8e435a/log.txt.gz 2025-11-16 22:02:26,682 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 22:02:27,013 - urllib3.connectionpool - DEBUG - https://files-staging.kernelci.org:443 "POST /upload HTTP/1.1" 200 0 2025-11-16 22:02:27,014 - lava_callback - INFO - Callback data uploaded to https://files-staging.kernelci.org/ltp-ipc-691873e2be9adb01fa8e435a/lava_callback.json.gz 2025-11-16 22:02:27,082 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 22:02:27,121 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691873e2be9adb01fa8e435a HTTP/1.1" 200 None 2025-11-16 22:02:27,122 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 22:02:27,163 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/69181a354a7acaa232fb6859 HTTP/1.1" 200 None 2025-11-16 22:02:27,165 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 22:02:27,200 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "PUT /latest/nodes/691873e2be9adb01fa8e435a HTTP/1.1" 200 None 2025-11-16 22:02:27,201 - lava_callback - INFO - Completed processing callback for node 691873e2be9adb01fa8e435a INFO: 172.21.0.1:60100 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36588 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36590 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:59870 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:36638 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:41864 - "GET /api/metrics HTTP/1.0" 200 OK INFO: 172.21.0.1:41874 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:49272 - "GET / HTTP/1.0" 200 OK INFO: 172.21.0.1:40120 - "GET / HTTP/1.0" 200 OK 2025-11-16 22:03:46,011 - lava_callback - INFO - Received callback for node 691873e3be9adb01fa8e435d INFO: 172.21.0.1:43018 - "POST /node/691873e3be9adb01fa8e435d HTTP/1.0" 202 Accepted 2025-11-16 22:03:46,022 - lava_callback - INFO - Starting async job submit for node 691873e3be9adb01fa8e435d 2025-11-16 22:03:46,108 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 22:03:46,126 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691873e3be9adb01fa8e435d HTTP/1.1" 200 None 2025-11-16 22:03:46,127 - lava_callback - DEBUG - Processing node 691873e3be9adb01fa8e435d: ltp-pty INFO: 172.21.0.1:43032 - "GET / HTTP/1.0" 200 OK 2025-11-16 22:03:46,781 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 2025-11-16 22:03:47,712 - lava_callback - INFO - Received callback for node 691873fabe9adb01fa8e436c INFO: 172.21.0.1:43048 - "POST /node/691873fabe9adb01fa8e436c HTTP/1.0" 202 Accepted 2025-11-16 22:03:47,724 - lava_callback - INFO - Starting async job submit for node 691873fabe9adb01fa8e436c 2025-11-16 22:03:47,768 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): staging.kernelci.org:9000 2025-11-16 22:03:47,784 - urllib3.connectionpool - DEBUG - https://staging.kernelci.org:9000 "GET /latest/node/691873fabe9adb01fa8e436c HTTP/1.1" 200 None 2025-11-16 22:03:47,784 - lava_callback - DEBUG - Processing node 691873fabe9adb01fa8e436c: ltp-timers INFO: 172.21.0.1:43050 - "GET / HTTP/1.0" 200 OK 2025-11-16 22:03:48,361 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): files-staging.kernelci.org:443 INFO: 172.21.0.1:37998 - "GET / HTTP/1.0" 200 OK INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [1]