01/06/2026 08:59:01 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/06/2026 08:59:02 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/06/2026 08:59:02 AM UTC [INFO] Runtime types from KContext: [] 01/06/2026 08:59:02 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/06/2026 08:59:02 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/06/2026 08:59:02 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/06/2026 08:59:02 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/06/2026 08:59:02 AM UTC [INFO] Default storage config from KContext: kci-storage 01/06/2026 08:59:02 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/06/2026 08:59:02 AM UTC [INFO] KContext get_storage_config returned: True 01/06/2026 08:59:02 AM UTC [INFO] KContext storage initialization successful: True 01/06/2026 08:59:02 AM UTC [INFO] Watchdog started with 600s timeout 01/06/2026 08:59:02 AM UTC [DEBUG] Node channel sub id: 102316 01/06/2026 08:59:02 AM UTC [DEBUG] Retry channel sub id: 102317 01/06/2026 08:59:02 AM UTC [INFO] Listening for available checkout events 01/06/2026 08:59:02 AM UTC [INFO] Press Ctrl-C to stop. 01/06/2026 08:59:02 AM UTC [INFO] Listening for available checkout events 01/06/2026 08:59:02 AM UTC [INFO] Press Ctrl-C to stop. 01/06/2026 08:59:09 AM UTC [DEBUG] Job node created: 695cceddc42011bf8acd6fad. Parent: 695cc06d9081037510059dd9 01/06/2026 08:59:10 AM UTC [INFO] LAVA queue status: device_type=mt8365-genio-350-evk queued=0 01/06/2026 08:59:11 AM UTC [INFO] 695cceddc42011bf8acd6fad lava-collabora mt8365-genio-350-evk baseline-nfs-arm64 21013599 127.0.0.1 - - [06/Jan/2026 08:59:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 08:59:54] "GET /health HTTP/1.1" 200 - 01/06/2026 09:00:18 AM UTC [DEBUG] Job node created: 695ccf22c42011bf8acd6fbb. Parent: 69585fe2e42e2c3506645845 01/06/2026 09:00:21 AM UTC [INFO] LAVA queue status: device_type=rk3399-khadas-edge-v queued=19 01/06/2026 09:00:21 AM UTC [INFO] 695ccf22c42011bf8acd6fbb lava-clabbe rk3399-khadas-edge-v baseline-arm64-clabbe 559292 127.0.0.1 - - [06/Jan/2026 09:00:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:00:54] "GET /health HTTP/1.1" 200 - 01/06/2026 09:00:59 AM UTC [DEBUG] Job node created: 695ccf4bc42011bf8acd6fc4. Parent: 695cc2bd908103751005a787 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260106 name: 5900003caf404c0eb85d030a2f61357f.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260106/5900003caf404c0eb85d030a2f61357f.json 01/06/2026 09:00:59 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260106/5900003caf404c0eb85d030a2f61357f.json 01/06/2026 09:00:59 AM UTC [INFO] 695ccf4bc42011bf8acd6fc4 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/06/2026 09:00:59 AM UTC [DEBUG] Job node created: 695ccf4bc42011bf8acd6fc7. Parent: 695cc2bd908103751005a787 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260106 name: 4bfdfa42e15d4bf4897f594d5a4de230.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260106/4bfdfa42e15d4bf4897f594d5a4de230.json 01/06/2026 09:01:00 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260106/4bfdfa42e15d4bf4897f594d5a4de230.json 01/06/2026 09:01:00 AM UTC [INFO] 695ccf4bc42011bf8acd6fc7 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 01/06/2026 09:01:08 AM UTC [DEBUG] Job node created: 695ccf54c42011bf8acd6fce. Parent: 695a21f22c9ee8be2a342664 01/06/2026 09:01:10 AM UTC [INFO] LAVA queue status: device_type=k3-j721e-beagleboneai64 queued=97 01/06/2026 09:01:10 AM UTC [INFO] 695ccf54c42011bf8acd6fce lava-baylibre k3-j721e-beagleboneai64 baseline-arm64-baylibre 4015095 01/06/2026 09:01:15 AM UTC [DEBUG] Job node created: 695ccf5bc42011bf8acd6fdf. Parent: 695a21f22c9ee8be2a342664 01/06/2026 09:01:15 AM UTC [INFO] LAVA device type k3-j721e-sk: no online devices 01/06/2026 09:01:15 AM UTC [INFO] 695ccf5bc42011bf8acd6fdf lava-baylibre k3-j721e-sk baseline-arm64-baylibre 4015096 01/06/2026 09:01:19 AM UTC [DEBUG] Job node created: 695ccf5fc42011bf8acd6fe2. Parent: 695a21f22c9ee8be2a342664 01/06/2026 09:01:20 AM UTC [INFO] LAVA device type meson-axg-s400: no online devices 01/06/2026 09:01:20 AM UTC [INFO] 695ccf5fc42011bf8acd6fe2 lava-baylibre meson-axg-s400 baseline-arm64-baylibre 4015097 01/06/2026 09:01:23 AM UTC [DEBUG] Job node created: 695ccf63c42011bf8acd6fe5. Parent: 695a21f22c9ee8be2a342664 127.0.0.1 - - [06/Jan/2026 09:01:24] "GET /health HTTP/1.1" 200 - 01/06/2026 09:01:25 AM UTC [INFO] LAVA queue status: device_type=meson-g12a-u200 queued=97 01/06/2026 09:01:25 AM UTC [INFO] 695ccf63c42011bf8acd6fe5 lava-baylibre meson-g12a-u200 baseline-arm64-baylibre 4015098 01/06/2026 09:01:29 AM UTC [DEBUG] Job node created: 695ccf69c42011bf8acd6ff2. Parent: 695a21f22c9ee8be2a342664 01/06/2026 09:01:31 AM UTC [INFO] LAVA queue status: device_type=meson-g12a-x96-max queued=96 01/06/2026 09:01:31 AM UTC [INFO] 695ccf69c42011bf8acd6ff2 lava-baylibre meson-g12a-x96-max baseline-arm64-baylibre 4015099 01/06/2026 09:01:35 AM UTC [DEBUG] Job node created: 695ccf6fc42011bf8acd6ff5. Parent: 695a21f22c9ee8be2a342664 01/06/2026 09:01:37 AM UTC [INFO] LAVA queue status: device_type=meson-gxl-s905x-libretech-cc queued=96 01/06/2026 09:01:37 AM UTC [INFO] 695ccf6fc42011bf8acd6ff5 lava-baylibre meson-gxl-s905x-libretech-cc baseline-arm64-baylibre 4015100 01/06/2026 09:01:42 AM UTC [DEBUG] Job node created: 695ccf76c42011bf8acd6ff8. Parent: 695a21f22c9ee8be2a342664 01/06/2026 09:01:44 AM UTC [INFO] LAVA queue status: device_type=meson-sm1-khadas-vim3l queued=97 01/06/2026 09:01:44 AM UTC [INFO] 695ccf76c42011bf8acd6ff8 lava-baylibre meson-sm1-khadas-vim3l baseline-arm64-baylibre 4015101 01/06/2026 09:01:49 AM UTC [DEBUG] Job node created: 695ccf7dc42011bf8acd6ffb. Parent: 695a21f22c9ee8be2a342664 01/06/2026 09:01:51 AM UTC [INFO] LAVA queue status: device_type=meson-sm1-sei610 queued=97 01/06/2026 09:01:51 AM UTC [INFO] 695ccf7dc42011bf8acd6ffb lava-baylibre meson-sm1-sei610 baseline-arm64-baylibre 4015102 127.0.0.1 - - [06/Jan/2026 09:01:55] "GET /health HTTP/1.1" 200 - 01/06/2026 09:01:57 AM UTC [DEBUG] Job node created: 695ccf85c42011bf8acd6ffe. Parent: 695a21f22c9ee8be2a342664 01/06/2026 09:01:58 AM UTC [INFO] LAVA device type sun50i-a64-pine64-plus: no online devices 01/06/2026 09:01:58 AM UTC [INFO] 695ccf85c42011bf8acd6ffe lava-baylibre sun50i-a64-pine64-plus baseline-arm64-baylibre 4015103 01/06/2026 09:02:03 AM UTC [DEBUG] Job node created: 695ccf8bc42011bf8acd7001. Parent: 695a21f22c9ee8be2a342664 127.0.0.1 - - [06/Jan/2026 09:02:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:02:55] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 534, in _make_request response = conn.getresponse() ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 516, in getresponse httplib_response = super().getresponse() ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/http/client.py", line 1430, in getresponse response.begin() File "/usr/local/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/http/client.py", line 292, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/socket.py", line 720, in readinto return self._sock.recv_into(b) ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/ssl.py", line 1251, in recv_into return self.read(nbytes, buffer) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/ssl.py", line 1103, in read return self._sslobj.read(len, buffer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TimeoutError: The read operation timed out The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 644, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 474, in increment raise reraise(type(error), error, _stacktrace) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/util.py", line 39, in reraise raise value File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 536, in _make_request self._raise_timeout(err=e, url=url, timeout_value=read_timeout) File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 367, in _raise_timeout raise ReadTimeoutError( urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files-staging.kernelci.org', port=443): Read timed out. (read timeout=60) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 701, in _run_scheduler self._run_job(job, runtime, platform, input_node, retry_counter) File "/home/kernelci/pipeline/./src/scheduler.py", line 421, in _run_job params = runtime.get_params(job, self._api.config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 352, in get_params params = super().get_params(job, api_config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/__init__.py", line 194, in get_params req = requests.get(metadata_url, timeout=60) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 73, in get return request("get", url, params=params, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 690, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='files-staging.kernelci.org', port=443): Read timed out. (read timeout=60) 127.0.0.1 - - [06/Jan/2026 09:03:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:03:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:04:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:04:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:05:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:05:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:06:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:06:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:07:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:07:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:08:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:08:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:09:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:09:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:10:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:10:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:11:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:11:56] "GET /health HTTP/1.1" 200 - 01/06/2026 09:12:02 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 601s (timeout: 600s). Forcing immediate exit! 01/06/2026 09:12:02 AM UTC [ERROR] ================================================================================ 01/06/2026 09:12:02 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/06/2026 09:12:02 AM UTC [ERROR] Thread: scheduler-retry, stuck for 601s 01/06/2026 09:12:02 AM UTC [ERROR] ================================================================================ 01/06/2026 08:59:01 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/06/2026 08:59:02 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/06/2026 08:59:02 AM UTC [INFO] Runtime types from KContext: [] 01/06/2026 08:59:02 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/06/2026 08:59:02 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/06/2026 08:59:02 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/06/2026 08:59:02 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/06/2026 08:59:02 AM UTC [INFO] Default storage config from KContext: kci-storage 01/06/2026 08:59:02 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/06/2026 08:59:02 AM UTC [INFO] KContext get_storage_config returned: True 01/06/2026 08:59:02 AM UTC [INFO] KContext storage initialization successful: True 01/06/2026 08:59:02 AM UTC [INFO] Watchdog started with 600s timeout 01/06/2026 08:59:02 AM UTC [DEBUG] Node channel sub id: 102316 01/06/2026 08:59:02 AM UTC [DEBUG] Retry channel sub id: 102317 01/06/2026 08:59:02 AM UTC [INFO] Listening for available checkout events 01/06/2026 08:59:02 AM UTC [INFO] Press Ctrl-C to stop. 01/06/2026 08:59:02 AM UTC [INFO] Listening for available checkout events 01/06/2026 08:59:02 AM UTC [INFO] Press Ctrl-C to stop. 01/06/2026 08:59:09 AM UTC [DEBUG] Job node created: 695cceddc42011bf8acd6fad. Parent: 695cc06d9081037510059dd9 01/06/2026 08:59:10 AM UTC [INFO] LAVA queue status: device_type=mt8365-genio-350-evk queued=0 01/06/2026 08:59:11 AM UTC [INFO] 695cceddc42011bf8acd6fad lava-collabora mt8365-genio-350-evk baseline-nfs-arm64 21013599 127.0.0.1 - - [06/Jan/2026 08:59:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 08:59:54] "GET /health HTTP/1.1" 200 - 01/06/2026 09:00:18 AM UTC [DEBUG] Job node created: 695ccf22c42011bf8acd6fbb. Parent: 69585fe2e42e2c3506645845 01/06/2026 09:00:21 AM UTC [INFO] LAVA queue status: device_type=rk3399-khadas-edge-v queued=19 01/06/2026 09:00:21 AM UTC [INFO] 695ccf22c42011bf8acd6fbb lava-clabbe rk3399-khadas-edge-v baseline-arm64-clabbe 559292 127.0.0.1 - - [06/Jan/2026 09:00:24] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:00:54] "GET /health HTTP/1.1" 200 - 01/06/2026 09:00:59 AM UTC [DEBUG] Job node created: 695ccf4bc42011bf8acd6fc4. Parent: 695cc2bd908103751005a787 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260106 name: 5900003caf404c0eb85d030a2f61357f.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260106/5900003caf404c0eb85d030a2f61357f.json 01/06/2026 09:00:59 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260106/5900003caf404c0eb85d030a2f61357f.json 01/06/2026 09:00:59 AM UTC [INFO] 695ccf4bc42011bf8acd6fc4 pull-labs-demo cd8180-orion-o6 ltp-smoketest-pull-labs 01/06/2026 09:00:59 AM UTC [DEBUG] Job node created: 695ccf4bc42011bf8acd6fc7. Parent: 695cc2bd908103751005a787 Storing PULL_LABS job definition to 'kci-storage' path: pull_labs_jobs/20260106 name: 4bfdfa42e15d4bf4897f594d5a4de230.json Job definition stored at URL: https://files-staging.kernelci.org/pull_labs_jobs/20260106/4bfdfa42e15d4bf4897f594d5a4de230.json 01/06/2026 09:01:00 AM UTC [DEBUG] Job definition URL: https://files-staging.kernelci.org/pull_labs_jobs/20260106/4bfdfa42e15d4bf4897f594d5a4de230.json 01/06/2026 09:01:00 AM UTC [INFO] 695ccf4bc42011bf8acd6fc7 pull-labs-demo qemu-arm64 ltp-smoketest-pull-labs 01/06/2026 09:01:08 AM UTC [DEBUG] Job node created: 695ccf54c42011bf8acd6fce. Parent: 695a21f22c9ee8be2a342664 01/06/2026 09:01:10 AM UTC [INFO] LAVA queue status: device_type=k3-j721e-beagleboneai64 queued=97 01/06/2026 09:01:10 AM UTC [INFO] 695ccf54c42011bf8acd6fce lava-baylibre k3-j721e-beagleboneai64 baseline-arm64-baylibre 4015095 01/06/2026 09:01:15 AM UTC [DEBUG] Job node created: 695ccf5bc42011bf8acd6fdf. Parent: 695a21f22c9ee8be2a342664 01/06/2026 09:01:15 AM UTC [INFO] LAVA device type k3-j721e-sk: no online devices 01/06/2026 09:01:15 AM UTC [INFO] 695ccf5bc42011bf8acd6fdf lava-baylibre k3-j721e-sk baseline-arm64-baylibre 4015096 01/06/2026 09:01:19 AM UTC [DEBUG] Job node created: 695ccf5fc42011bf8acd6fe2. Parent: 695a21f22c9ee8be2a342664 01/06/2026 09:01:20 AM UTC [INFO] LAVA device type meson-axg-s400: no online devices 01/06/2026 09:01:20 AM UTC [INFO] 695ccf5fc42011bf8acd6fe2 lava-baylibre meson-axg-s400 baseline-arm64-baylibre 4015097 01/06/2026 09:01:23 AM UTC [DEBUG] Job node created: 695ccf63c42011bf8acd6fe5. Parent: 695a21f22c9ee8be2a342664 127.0.0.1 - - [06/Jan/2026 09:01:24] "GET /health HTTP/1.1" 200 - 01/06/2026 09:01:25 AM UTC [INFO] LAVA queue status: device_type=meson-g12a-u200 queued=97 01/06/2026 09:01:25 AM UTC [INFO] 695ccf63c42011bf8acd6fe5 lava-baylibre meson-g12a-u200 baseline-arm64-baylibre 4015098 01/06/2026 09:01:29 AM UTC [DEBUG] Job node created: 695ccf69c42011bf8acd6ff2. Parent: 695a21f22c9ee8be2a342664 01/06/2026 09:01:31 AM UTC [INFO] LAVA queue status: device_type=meson-g12a-x96-max queued=96 01/06/2026 09:01:31 AM UTC [INFO] 695ccf69c42011bf8acd6ff2 lava-baylibre meson-g12a-x96-max baseline-arm64-baylibre 4015099 01/06/2026 09:01:35 AM UTC [DEBUG] Job node created: 695ccf6fc42011bf8acd6ff5. Parent: 695a21f22c9ee8be2a342664 01/06/2026 09:01:37 AM UTC [INFO] LAVA queue status: device_type=meson-gxl-s905x-libretech-cc queued=96 01/06/2026 09:01:37 AM UTC [INFO] 695ccf6fc42011bf8acd6ff5 lava-baylibre meson-gxl-s905x-libretech-cc baseline-arm64-baylibre 4015100 01/06/2026 09:01:42 AM UTC [DEBUG] Job node created: 695ccf76c42011bf8acd6ff8. Parent: 695a21f22c9ee8be2a342664 01/06/2026 09:01:44 AM UTC [INFO] LAVA queue status: device_type=meson-sm1-khadas-vim3l queued=97 01/06/2026 09:01:44 AM UTC [INFO] 695ccf76c42011bf8acd6ff8 lava-baylibre meson-sm1-khadas-vim3l baseline-arm64-baylibre 4015101 01/06/2026 09:01:49 AM UTC [DEBUG] Job node created: 695ccf7dc42011bf8acd6ffb. Parent: 695a21f22c9ee8be2a342664 01/06/2026 09:01:51 AM UTC [INFO] LAVA queue status: device_type=meson-sm1-sei610 queued=97 01/06/2026 09:01:51 AM UTC [INFO] 695ccf7dc42011bf8acd6ffb lava-baylibre meson-sm1-sei610 baseline-arm64-baylibre 4015102 127.0.0.1 - - [06/Jan/2026 09:01:55] "GET /health HTTP/1.1" 200 - 01/06/2026 09:01:57 AM UTC [DEBUG] Job node created: 695ccf85c42011bf8acd6ffe. Parent: 695a21f22c9ee8be2a342664 01/06/2026 09:01:58 AM UTC [INFO] LAVA device type sun50i-a64-pine64-plus: no online devices 01/06/2026 09:01:58 AM UTC [INFO] 695ccf85c42011bf8acd6ffe lava-baylibre sun50i-a64-pine64-plus baseline-arm64-baylibre 4015103 01/06/2026 09:02:03 AM UTC [DEBUG] Job node created: 695ccf8bc42011bf8acd7001. Parent: 695a21f22c9ee8be2a342664 127.0.0.1 - - [06/Jan/2026 09:02:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:02:55] "GET /health HTTP/1.1" 200 - Exception in thread scheduler-retry: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 534, in _make_request response = conn.getresponse() ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connection.py", line 516, in getresponse httplib_response = super().getresponse() ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/http/client.py", line 1430, in getresponse response.begin() File "/usr/local/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/http/client.py", line 292, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/socket.py", line 720, in readinto return self._sock.recv_into(b) ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/ssl.py", line 1251, in recv_into return self.read(nbytes, buffer) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/ssl.py", line 1103, in read return self._sslobj.read(len, buffer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TimeoutError: The read operation timed out The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 644, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 841, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/retry.py", line 474, in increment raise reraise(type(error), error, _stacktrace) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/util/util.py", line 39, in reraise raise value File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 787, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 536, in _make_request self._raise_timeout(err=e, url=url, timeout_value=read_timeout) File "/usr/local/lib/python3.12/site-packages/urllib3/connectionpool.py", line 367, in _raise_timeout raise ReadTimeoutError( urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files-staging.kernelci.org', port=443): Read timed out. (read timeout=60) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run() File "/usr/local/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs) File "/home/kernelci/pipeline/./src/scheduler.py", line 701, in _run_scheduler self._run_job(job, runtime, platform, input_node, retry_counter) File "/home/kernelci/pipeline/./src/scheduler.py", line 421, in _run_job params = runtime.get_params(job, self._api.config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/lava.py", line 352, in get_params params = super().get_params(job, api_config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/kernelci/runtime/__init__.py", line 194, in get_params req = requests.get(metadata_url, timeout=60) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 73, in get return request("get", url, params=params, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/requests/adapters.py", line 690, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='files-staging.kernelci.org', port=443): Read timed out. (read timeout=60) 127.0.0.1 - - [06/Jan/2026 09:03:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:03:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:04:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:04:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:05:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:05:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:06:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:06:55] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:07:25] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:07:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:08:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:08:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:09:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:09:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:10:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:10:56] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:11:26] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:11:56] "GET /health HTTP/1.1" 200 - 01/06/2026 09:12:02 AM UTC [ERROR] WATCHDOG: Thread 'scheduler-retry' stuck for 601s (timeout: 600s). Forcing immediate exit! 01/06/2026 09:12:02 AM UTC [ERROR] ================================================================================ 01/06/2026 09:12:02 AM UTC [ERROR] STUCK THREAD DETECTED - FORCING EXIT 01/06/2026 09:12:02 AM UTC [ERROR] Thread: scheduler-retry, stuck for 601s 01/06/2026 09:12:02 AM UTC [ERROR] ================================================================================ 01/06/2026 09:12:04 AM UTC [INFO] Job backup disabled (set BACKUP_FILE_LIFETIME env var to enable) 01/06/2026 09:12:05 AM UTC [INFO] Runtimes from KContext: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/06/2026 09:12:05 AM UTC [INFO] Runtime types from KContext: [] 01/06/2026 09:12:05 AM UTC [INFO] Initializing runtimes: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/06/2026 09:12:05 AM UTC [INFO] Filtering runtimes by name: ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-qualcomm', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/06/2026 09:12:05 AM UTC [WARNING] Runtime 'lava-qualcomm' not found in configuration 01/06/2026 09:12:05 AM UTC [INFO] Selected 11 runtime(s): ['lava-collabora', 'lava-collabora-staging', 'lava-clabbe', 'lava-broonie', 'lava-baylibre', 'lava-cip', 'lava-pengutronix', 'lava-kontron', 'lava-foundriesio', 'lava-testpull', 'pull-labs-demo'] 01/06/2026 09:12:05 AM UTC [INFO] Default storage config from KContext: kci-storage 01/06/2026 09:12:05 AM UTC [INFO] Attempting to initialize storage config: kci-storage 01/06/2026 09:12:05 AM UTC [INFO] KContext get_storage_config returned: True 01/06/2026 09:12:05 AM UTC [INFO] KContext storage initialization successful: True 01/06/2026 09:12:05 AM UTC [INFO] Watchdog started with 600s timeout 01/06/2026 09:12:05 AM UTC [DEBUG] Node channel sub id: 102322 01/06/2026 09:12:05 AM UTC [DEBUG] Retry channel sub id: 102323 01/06/2026 09:12:05 AM UTC [INFO] Listening for available checkout events 01/06/2026 09:12:05 AM UTC [INFO] Press Ctrl-C to stop. 01/06/2026 09:12:05 AM UTC [INFO] Listening for available checkout events 01/06/2026 09:12:05 AM UTC [INFO] Press Ctrl-C to stop. 01/06/2026 09:12:27 AM UTC [DEBUG] Job node created: 695cd1fbc42011bf8acd7166. Parent: 69585fcee42e2c3506645836 01/06/2026 09:12:30 AM UTC [INFO] LAVA queue status: device_type=sun7i-a20-cubieboard2 queued=19 01/06/2026 09:12:30 AM UTC [INFO] 695cd1fbc42011bf8acd7166 lava-clabbe sun7i-a20-cubieboard2 baseline-arm-clabbe 559293 127.0.0.1 - - [06/Jan/2026 09:12:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:13:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:13:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:14:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:14:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:15:03] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:15:33] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:16:03] "GET /health HTTP/1.1" 200 - 01/06/2026 09:16:05 AM UTC [DEBUG] Job node created: 695cd2d4c42011bf8acd7178. Parent: 695cc06d9081037510059dd9 01/06/2026 09:16:06 AM UTC [INFO] LAVA queue status: device_type=mt8365-genio-350-evk queued=0 01/06/2026 09:16:06 AM UTC [INFO] 695cd2d4c42011bf8acd7178 lava-collabora mt8365-genio-350-evk baseline-nfs-arm64 21013787 127.0.0.1 - - [06/Jan/2026 09:16:34] "GET /health HTTP/1.1" 200 - 01/06/2026 09:16:44 AM UTC [DEBUG] Job node created: 695cd2fcc42011bf8acd717d. Parent: 695ccffcc42011bf8acd7036 01/06/2026 09:16:46 AM UTC [INFO] LAVA queue status: device_type=at91sam9g20ek queued=112 01/06/2026 09:16:47 AM UTC [INFO] 695cd2fcc42011bf8acd717d lava-broonie at91sam9g20ek baseline-arm 2336447 01/06/2026 09:16:47 AM UTC [DEBUG] Job node created: 695cd2ffc42011bf8acd7180. Parent: 695ccffcc42011bf8acd7036 01/06/2026 09:16:48 AM UTC [INFO] LAVA queue status: device_type=at91sam9g20ek queued=113 01/06/2026 09:16:48 AM UTC [INFO] 695cd2ffc42011bf8acd7180 lava-broonie at91sam9g20ek ltp-smoketest 2336448 01/06/2026 09:17:02 AM UTC [DEBUG] Job node created: 695cd30ec42011bf8acd7189. Parent: 695adf0c872d7868c4cbed2e 127.0.0.1 - - [06/Jan/2026 09:17:04] "GET /health HTTP/1.1" 200 - 01/06/2026 09:17:04 AM UTC [INFO] LAVA queue status: device_type=sun7i-a20-cubieboard2 queued=19 01/06/2026 09:17:04 AM UTC [INFO] 695cd30ec42011bf8acd7189 lava-clabbe sun7i-a20-cubieboard2 baseline-arm-clabbe 559294 01/06/2026 09:17:06 AM UTC [DEBUG] Job node created: 695cd312c42011bf8acd718c. Parent: 6954dbf40db87e6a5b1c0fa6 01/06/2026 09:17:07 AM UTC [INFO] LAVA queue status: device_type=rk3288-rock2-square queued=216 01/06/2026 09:17:07 AM UTC [INFO] 695cd312c42011bf8acd718c lava-collabora rk3288-rock2-square baseline-arm 21013789 127.0.0.1 - - [06/Jan/2026 09:17:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:18:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:18:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:19:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:19:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:20:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:20:34] "GET /health HTTP/1.1" 200 - 01/06/2026 09:20:56 AM UTC [DEBUG] Job node created: 695cd3f8c42011bf8acd71ba. Parent: 695adf1b872d7868c4cbed47 01/06/2026 09:20:58 AM UTC [INFO] LAVA queue status: device_type=rk3399-khadas-edge-v queued=19 01/06/2026 09:20:58 AM UTC [INFO] 695cd3f8c42011bf8acd71ba lava-clabbe rk3399-khadas-edge-v baseline-arm64-clabbe 559295 127.0.0.1 - - [06/Jan/2026 09:21:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:21:34] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:22:04] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:22:35] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:23:05] "GET /health HTTP/1.1" 200 - 127.0.0.1 - - [06/Jan/2026 09:23:35] "GET /health HTTP/1.1" 200 - 01/06/2026 09:23:36 AM UTC [INFO] Stopping. 01/06/2026 09:23:36 AM UTC [INFO] Unsubscribing: 102322 01/06/2026 09:23:36 AM UTC [INFO] Unsubscribing: 102323